Performance¶
Every number here comes from pyperf on CPython 3.14.6 (a release build) on an Apple
M4 running macOS 26. pyperf runs each case in isolated worker processes and reports the mean; the harness prints the
run-to-run standard deviation beside it as ±N% so a real gap reads apart from noise, and these tables quote the
mean. For the lowest-noise figures, tune the machine with pyperf system tune first (and sudo pyperf system reset
after); pyperf options like --rigorous or --affinity pass straight through after the tox -e bench command.
Operations that mutate the tree they are handed – the edits, the content setters, link absolutization – are timed on a
fresh parse rebuilt before each iteration (the rebuild itself untimed), so the figure is the repeatable cost of the
mutation alone rather than a tree a prior iteration already changed; read-path operations reuse one cached parse and
time only the query. The corpora are real documents: Project Gutenberg’s War and Peace, the WHATWG HTML specification source, the ECMAScript specification,
a size-weighted sample of web-platform-tests pages for the parse and
tokenize suites, and, for the read-path suites, real saved web pages – a blog, a news article, and a product blog from
the mozilla/readability test corpus – so the selector, link, and edit
operations run against genuine nested structure rather than the layout fixtures, which carry none. The harness
benchmarks each competitor in its own isolated uv venv – turbohtml in a venv of its own as the shared baseline –
so one library’s dependency pins never perturb another’s. Every table below is one harness operation, so each is
reproducible with tox -e bench <command>, where the command is core (turbohtml’s own baseline for every
operation), an operation name (the cross-competitor table), a package name (that competitor’s own report), or all.
By default the turbohtml baseline is a plain wheel, which builds quickly for iterating; pass --pgo before the
command (tox -e bench -- --pgo core) to build it instead with the shipped profile-guided, link-time-optimized
release recipe, so the baseline reads as what a release ships rather than a plain build, at the cost of a much slower
build. Most operations are a single call; a few aggregate workloads (build, build-e) sweep a size, and the
construct and emit breakdowns decompose that write path into the constructor and the serializer in isolation.
Numbers vary with input and hardware.
Escaping¶
turbohtml.escape() against the standard library’s html.escape(). It gains the most on text that needs
little escaping, where the SIMD scan classifies sixteen bytes at a time and copies clean stretches in bulk; the gap
narrows on tiny strings, where call overhead dominates.
input |
turbohtml |
|
|---|---|---|
tiny plain (64 B) |
60 ns |
120 ns (2.0x) |
medium markup (4 KiB) |
2.21 µs |
7.31 µs (3.4x) |
no-op prose (4 MiB) |
120 µs |
2.54 ms (21.2x) |
book text (3 MiB) |
670 µs |
2.65 ms (4.0x) |
book HTML (4 MiB) |
1.22 ms |
4.74 ms (3.9x) |
spec HTML, dense (4 MiB) |
4.92 ms |
13 ms (2.7x) |
UCS-2 plain (4 MiB) |
800 µs |
2.62 ms (3.3x) |
UCS-2 markup (4 MiB) |
5.7 ms |
11.2 ms (2.0x) |
UCS-4 plain (4 MiB) |
920 µs |
5.73 ms (6.3x) |
UCS-4 markup (4 MiB) |
6.69 ms |
20 ms (3.0x) |
Markup (escaping)¶
turbohtml.migration.markupsafe.escape() against markupsafe’s own C
escape, both returning a Markup. The inputs are the small, mostly-clean strings a template engine interpolates under
autoescape, markupsafe’s hottest path. turbohtml builds the safe string in C in a single call, where markupsafe pays a
Python escape frame and Markup construction per call, so it runs roughly two to three times faster.
input |
turbohtml |
|
|---|---|---|
clean (8 B) |
60 ns |
203 ns (3.4x) |
clean (32 B) |
70 ns |
223 ns (3.2x) |
clean (256 B) |
134 ns |
477 ns (3.6x) |
name with |
88 ns |
230 ns (2.7x) |
escape-heavy markup |
153 ns |
365 ns (2.4x) |
The other Markup operations race markupsafe’s own Markup of the same method. striptags and unescape run
on turbohtml’s tokenizer and HTML5 reference resolution where markupsafe scans with a regex, and format and join
escape each untrusted operand through the same C escape.
operation |
turbohtml |
|
|---|---|---|
|
1.21 µs |
2.37 µs (2.0x) |
|
223 ns |
1.04 µs (4.7x) |
|
1.8 µs |
2.09 µs (1.2x) |
|
671 ns |
1.22 µs (1.9x) |
Linkify¶
turbohtml.clean.linkify() against bleach’s linkify, the HTML-aware
linkifier it succeeds, and linkify-it-py, the pure-Python scanner
markdown-it-py pulls in. bleach and turbohtml both parse the HTML and rewrite it; linkify-it-py only finds the matches
and does not rewrite, so it does strictly less work, yet turbohtml is faster than both. The C candidate scan and
turbohtml’s own tree carry it past bleach’s html5lib pass by five to twenty times.
input |
turbohtml |
||
|---|---|---|---|
comment (1 link, 1 email) |
3.1 µs |
31 µs (10.0x) |
57 µs (18.4x) |
prose (1 KiB) |
52 µs |
330 µs (6.4x) |
293 µs (5.7x) |
markup (4 KiB) |
130 µs |
800 µs (6.2x) |
1.68 ms (13.0x) |
The detection primitive on its own, turbohtml.clean.LinkDetector.find() against LinkifyIt().match and
has_link() against LinkifyIt().test, scans a run of plain text and returns the
spans or a boolean without rewriting any HTML, so this isolates the C scan from the full linkify rewrite above. The
has_link prose row is close because test short-circuits on the first link near the start.
detect |
turbohtml |
|
|---|---|---|
|
700 ns |
30 µs (42.9x) |
|
8.9 µs |
316 µs (35.6x) |
|
300 ns |
22.4 µs (74.7x) |
|
2.5 µs |
4.8 µs (2.0x) |
Sanitize¶
turbohtml.clean.sanitize() against four sanitizers. Three share its allowlist model, where only listed tags and
attributes survive, so a vector nobody anticipated is dropped by default: nh3 (the Rust
ammonia binding), bleach (its end-of-life predecessor, on html5lib), and
html-sanitizer (an allowlist over lxml). The fourth, lxml-html-clean (the externalized lxml.html.clean.Cleaner), is a blocklist: it
strips the constructs it knows are dangerous and lets the rest through, a model lxml itself flagged as hard to keep
safe. The inputs are realistic user content with a few disallowed tags and a dangerous attribute mixed in. turbohtml
runs the whole filtering walk in C and leads every alternative, but the model matters more than the microseconds. Prefer
an allowlist, since a blocklist passes anything it did not think to name.
input |
turbohtml |
||||
|---|---|---|---|---|---|
comment (1 link, 1 script) |
1.6 µs |
5.8 µs (3.7x) |
20.4 µs (12.8x) |
49.2 µs (30.8x) |
87.8 µs (54.9x) |
post (4 KiB) |
42.5 µs |
142 µs (3.4x) |
530 µs (12.5x) |
1.69 ms (39.8x) |
2.18 ms (51.5x) |
Markdown¶
turbohtml.Node.to_markdown() against markdownify (on
BeautifulSoup) and html2text (a streaming HTMLParser subclass). All three
take an HTML string and return Markdown, so each parses first; turbohtml parses to the WHATWG tree and walks it in C,
where the others build and convert in Python. The single C pass converts a page in tens of microseconds, two orders of
magnitude ahead of markdownify. The configured row turns the option surface on in all three (underscore emphasis,
reference links, padded tables, full escaping), and turbohtml stays ahead by the same margin.
input |
turbohtml |
||
|---|---|---|---|
article (2 KiB) |
2.8 µs |
595 µs (213x) |
1.28 ms (458x) |
list (4 KiB) |
4.5 µs |
1.25 ms (279x) |
2.6 ms (577x) |
table (4 KiB) |
7.3 µs |
1.1 ms (151x) |
3.07 ms (421x) |
configured (4 KiB) |
15 µs |
1.23 ms (82.2x) |
2.77 ms (185x) |
google_doc (4 KiB) |
11 µs |
600 µs (54.6x) |
— |
The google_doc row reads the inline-CSS styling a Google Docs export carries (html2text’s google_doc mode);
markdownify has no equivalent.
Structured data¶
turbohtml.Document.structured_data() against extruct, the scraper
toolkit it succeeds, extracting JSON-LD, Microdata, and OpenGraph from a product page that carries all three. Both start
from the raw HTML string, so each parses first; extruct builds an lxml tree and runs a separate extractor per syntax,
where turbohtml parses to the WHATWG tree and gathers every format in one C walk, handing back the typed
StructuredData record. The single pass runs roughly nine to eleven times faster.
input |
turbohtml |
|
|---|---|---|
product page |
6.2 µs |
60 µs (9.7x) |
catalog (8 KiB) |
56.3 µs |
560 µs (10.0x) |
Tables¶
turbohtml.Node.tables() and turbohtml.Element.records() against pandas’s
read_html, the one-call table reader scrapers reach for. Both parse the HTML and extract every <table>,
resolving rowspan and colspan into a rectangular grid; read_html returns a DataFrame per table and pulls
in NumPy, where turbohtml runs the cell-grid walk in C and hands back plain list and dict objects with no added
dependency. The rows row times tables() (every table as list[list[str]]) and the
records row times records() (the first table keyed by its header), each over a four-column
table of the given height. The single C pass leads from roughly twelve times on the largest table to nearly ninety on
the smallest, where pandas pays its fixed per-frame construction cost.
input |
turbohtml |
|
|---|---|---|
rows (10 rows) |
2.8 µs |
379 µs (136x) |
records (10 rows) |
2.89 µs |
480 µs (167x) |
rows (100 rows) |
24.8 µs |
1.06 ms (42.9x) |
records (100 rows) |
26.1 µs |
1.23 ms (47.2x) |
rows (1000 rows) |
263 µs |
8.31 ms (31.6x) |
records (1000 rows) |
275 µs |
9.1 ms (33.1x) |
Article extraction¶
turbohtml.Node.article() against trafilatura, readability-lxml, newspaper3k, goose3, readabilipy, and news-please, the article extractors it succeeds. Each scores the dominant content body
and (trafilatura, newspaper3k, goose3, and news-please) harvests the page metadata beside it; the lxml-backed four build
their tree in Python first, readabilipy’s Python mode parses with html5lib into BeautifulSoup and cleans without
scoring, news-please merges the votes of several such extractors, and turbohtml does the scoring and the harvest in one
C pass over the parsed tree. The inputs are full pages – navigation, a scored article, and a footer – so the
boilerplate the heuristic discounts is part of the measured cost.
input |
turbohtml |
||||||
|---|---|---|---|---|---|---|---|
post (4 KiB) |
6.9 µs |
600 µs (87.0x) |
640 µs (92.8x) |
2.08 ms (302x) |
2.29 ms (332x) |
5.6 ms (812x) |
5 ms (725x) |
longform (16 KiB) |
26 µs |
1.64 ms (63.1x) |
1.9 ms (73.1x) |
5.8 ms (224x) |
8.34 ms (321x) |
15.3 ms (589x) |
18.3 ms (704x) |
Boilerplate classification¶
turbohtml.extract.boilerplate() against justext and boilerpy3, the per-block boilerplate classifiers. All three segment the page into units
and mark each good or boilerplate; justext scores every paragraph in Python over an lxml tree (length, link density,
stopword density), boilerpy3 classifies the blocks of its own SAX stream with boilerpipe’s rules, and turbohtml scores
the tree once in C and classifies the units in a thin Python layer. The inputs are the article-extraction pages, so the
navigation and footer each classifier must reject are part of the measured cost.
Date extraction¶
turbohtml.extract.dates() against htmldate, the standalone publication-date
finder. Both read the same signals – publication/modification <meta> tags, JSON-LD, <time> elements, and a date
in the URL – and both are parse-bound; htmldate builds an lxml tree, turbohtml the WHATWG tree. On the clean metadata
pages below htmldate’s header lookup returns first, so it edges ahead. On boilerplate pages with no date metadata the
gap inverts: htmldate prunes the tree and runs its dateparser-backed text scoring, where turbohtml’s early-exit over the
structured signals runs 2x-4x faster (a 149 kB page: 2.9 ms against 12.2 ms).
publication-date extraction |
turbohtml |
|
|---|---|---|
post (4 KiB) |
9.15 µs |
17.6 µs (2.0x) |
longform (16 KiB) |
13.4 µs |
36.2 µs (2.8x) |
Unescaping¶
turbohtml.unescape() against html.unescape() and w3lib’s
replace_entities, the Scrapy helper that resolves the same references. It gains the most on entity-heavy input,
where the standard library pays a Python call per match and w3lib runs a regular-expression substitution with a Python
callback per match; turbohtml hops between & occurrences in C and bulk-copies the clean spans between references.
input |
turbohtml |
||
|---|---|---|---|
tiny plain (64 B) |
30 ns |
40 ns (1.4x) |
260 ns (8.7x) |
medium dense refs (4 KiB) |
7.1 µs |
70.7 µs (10.0x) |
114 µs (16.1x) |
numeric refs (4 KiB) |
4.97 µs |
79.2 µs (16.0x) |
90.9 µs (18.3x) |
book HTML, real refs (4 MiB) |
3.34 ms |
8.72 ms (2.7x) |
13.6 ms (4.1x) |
escaped book HTML (5 MiB) |
1.76 ms |
30 ms (17.1x) |
35 ms (19.9x) |
dense refs (4 MiB) |
9.67 ms |
85.7 ms (8.9x) |
121 ms (12.6x) |
UCS-2 refs (4 MiB) |
2.75 ms |
19.1 ms (7.0x) |
27.8 ms (10.2x) |
Tokenizing¶
turbohtml.tokenize() against html.parser.HTMLParser (driven with no-op handlers) and html5lib’s pure-Python tokenizer. The closest case is a document dominated by a single text
node, where the standard library’s regex performs one C scan; wherever markup appears, the state machine is roughly ten
times faster.
input |
turbohtml |
||
|---|---|---|---|
typical markup |
29 µs |
441 µs (15.3x) |
817 µs (28.2x) |
text-heavy prose |
540 ns |
2.7 µs (5.0x) |
145 µs (269x) |
attribute-heavy |
18.2 µs |
300 µs (16.5x) |
805 µs (44.3x) |
script-heavy |
11.5 µs |
152 µs (13.3x) |
500 µs (43.5x) |
entity-heavy |
19.8 µs |
192 µs (9.7x) |
1.2 ms (60.7x) |
wpt page (0.6 kB) |
1.52 µs |
18.4 µs (12.2x) |
47.6 µs (31.4x) |
wpt page (4 kB) |
12 µs |
167 µs (14.0x) |
423 µs (35.3x) |
wpt page (9.6 kB) |
27.3 µs |
387 µs (14.2x) |
1.16 ms (42.5x) |
wpt page (92 kB) |
314 µs |
4 ms (12.8x) |
9.03 ms (28.8x) |
wpt page, CJK (124 kB) |
550 µs |
8.58 ms (15.6x) |
22.3 ms (40.6x) |
whatwg spec (235 kB) |
624 µs |
7.4 ms (11.9x) |
19.7 ms (31.6x) |
ecmascript spec (3 MB) |
6.2 ms |
53.9 ms (8.7x) |
180 ms (29.1x) |
whatwg spec source (7.9 MB) |
36.5 ms |
370 ms (10.2x) |
850 ms (23.3x) |
Parsing¶
turbohtml.parse() builds a full WHATWG document tree, against the other Python tree builders: lxml, selectolax and resiliparse (both wrapping lexbor), BeautifulSoup over html.parser, and html5lib. turbohtml runs on par
with resiliparse, two to four times faster than lxml and selectolax, and 30 to 80 times faster than the pure-Python
builders, while building the WHATWG tree that lxml’s libxml2 does not.
resiliparse reaches turbohtml’s throughput because its HTMLTree.parse is a thin call straight into lexbor’s native
tree, while selectolax wraps that same engine behind a heavier object layer; the comparison here is parsing only.
resiliparse’s wider toolkit, boilerplate and main-content extraction, language detection, and the encoding and archive
utilities it ships for large-scale web-crawl processing, sits outside turbohtml’s scope. gumbo, the C WHATWG parser Google released and the engine behind html5-parser, has no row: it is read-oriented, archived upstream, and its Python binding no
longer builds on a current toolchain. turbohtml is the maintained, mutable, typed alternative to that lineage.
input |
turbohtml |
|||||
|---|---|---|---|---|---|---|
wpt page (0.6 kB) |
1.2 µs |
3.8 µs (3.2x) |
3.4 µs (2.9x) |
7.1 µs (6.0x) |
60.8 µs (50.7x) |
104 µs (86.7x) |
wpt page (4 kB) |
9.57 µs |
12.6 µs (1.4x) |
27 µs (2.9x) |
42.6 µs (4.5x) |
440 µs (46.0x) |
690 µs (72.2x) |
wpt page (9.6 kB) |
24.2 µs |
27.9 µs (1.2x) |
74 µs (3.1x) |
108 µs (4.5x) |
853 µs (35.3x) |
1.47 ms (60.8x) |
wpt page (92 kB) |
207 µs |
286 µs (1.4x) |
645 µs (3.2x) |
938 µs (4.6x) |
15 ms (72.5x) |
17 ms (82.2x) |
wpt page, CJK (124 kB) |
410 µs |
544 µs (1.4x) |
1.4 ms (3.5x) |
2.3 ms (5.7x) |
21.9 ms (53.5x) |
30.5 ms (74.4x) |
whatwg spec (235 kB) |
405 µs |
507 µs (1.3x) |
1.27 ms (3.2x) |
1.79 ms (4.5x) |
25 ms (61.8x) |
33.1 ms (81.8x) |
Fragment parsing¶
turbohtml.parse_fragment() parses an innerHTML-style snippet in a container’s context rather than a whole
document, against lxml’s lxml.html.fromstring and html5lib’s parseFragment. The input is a table-row fragment
parsed in its <tbody> context, where the WHATWG algorithm’s table rules apply. turbohtml runs the same C engine it
uses for whole documents, so it parses the fragment three times faster than lxml and roughly seventy times faster than
the pure-Python html5lib.
Querying¶
Each library parses the document once, then the timed call runs one query. find collects every <a> element the
way each library reaches for it (turbohtml’s find_all(), lxml’s XPath findall, selectolax’s
and BeautifulSoup’s selectors). A tag-only query resolves the name to an interned atom and walks the subtree comparing
integers, with no per-element string built and no matcher dispatch, so it runs ahead of lxml’s C XPath engine and many
times ahead of selectolax, parsel (Scrapy’s cssselect-over-libxml2 selector library), and BeautifulSoup.
find every <a> |
turbohtml |
||||
|---|---|---|---|---|---|
daring fireball (10 kB) |
400 ns |
5.1 µs (12.8x) |
5.9 µs (14.8x) |
21.1 µs (52.8x) |
13.5 µs (33.8x) |
ars technica (56 kB) |
800 ns |
13.4 µs (16.8x) |
15.8 µs (19.8x) |
47.9 µs (59.9x) |
50.5 µs (63.2x) |
mozilla blog (95 kB) |
1.2 µs |
20.4 µs (17.0x) |
28.9 µs (24.1x) |
71.6 µs (59.7x) |
107 µs (89.2x) |
whatwg spec (235 kB) |
1.3 µs |
44.8 µs (34.5x) |
75 µs (57.7x) |
105 µs (80.4x) |
393 µs (303x) |
select runs the CSS selector div a[href] (turbohtml’s select(), lxml’s cssselect, selectolax’s css, parsel’s css, BeautifulSoup’s soupsieve). Because turbohtml compiles the selector against the tree once and then
matches by comparing interned integer atoms, it stays in the low microseconds across these pages. lxml and parsel
re-translate the selector to XPath through cssselect on every call, which scales with the document and trails by tens of
times on the small blog up to roughly seven hundred times on the spec; BeautifulSoup’s soupsieve is hundreds to more
than fifteen hundred times behind, while selectolax, the other compiled engine, stays closest at roughly eleven to forty
times.
select |
turbohtml |
||||
|---|---|---|---|---|---|
daring fireball (10 kB) |
600 ns |
7.6 µs (12.7x) |
29.4 µs (49.0x) |
30 µs (50.0x) |
168 µs (281x) |
ars technica (56 kB) |
1.5 µs |
19.9 µs (13.3x) |
124 µs (82.7x) |
146 µs (97.2x) |
566 µs (378x) |
mozilla blog (95 kB) |
2.1 µs |
33.6 µs (16.0x) |
803 µs (383x) |
828 µs (395x) |
1.1 ms (524x) |
whatwg spec (235 kB) |
1.7 µs |
78.8 µs (46.4x) |
1.34 ms (789x) |
1.41 ms (830x) |
2.74 ms (1612x) |
The relational :has() pseudo-class is the costliest selector to evaluate, since a naive matcher rescans each
candidate’s subtree. turbohtml runs div:has(a) against the same pages and leads every alternative: tens of times
faster than lxml and selectolax on the smaller pages, narrowing to single digits on the link-dense mozilla blog where
the relational match itself does real work, while BeautifulSoup trails by hundreds of times throughout. The matcher
walks each anchor’s descendants once and skips the sibling scan for descendant and child relationships, so the
relational lookup keeps the same interned-atom comparison the flat selectors use.
select |
turbohtml |
|||
|---|---|---|---|---|
daring fireball (10 kB) |
300 ns |
5 µs (16.7x) |
14.3 µs (47.7x) |
112 µs (374x) |
ars technica (56 kB) |
1.3 µs |
18 µs (13.9x) |
27.7 µs (21.4x) |
490 µs (377x) |
mozilla blog (95 kB) |
9 µs |
52.9 µs (5.9x) |
60.5 µs (6.8x) |
2.2 ms (245x) |
whatwg spec (235 kB) |
6.1 µs |
86.1 µs (14.2x) |
71.2 µs (11.7x) |
3.2 ms (525x) |
Per-element matching runs each anchor on the page through a compiled div a[href] matcher – the shape a soupsieve
port hits through turbohtml.query and its Matcher.match (soupsieve is the
only competitor with a compiled per-element match). turbohtml answers each test with the same interned-atom comparison
its select uses, walking the ancestor chain once per candidate, where soupsieve re-interprets the parsed selector in
Python per element, so the sweep runs 94 to 155 times faster across these pages.
match each anchor against div a[href] |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
1.89 µs |
170 µs (90.0x) |
ars technica (56 kB) |
4.28 µs |
782 µs (183x) |
mozilla blog (95 kB) |
6 µs |
1.08 ms (180x) |
whatwg spec (235 kB) |
7.3 µs |
1.25 ms (172x) |
A text-content search runs through find_all() with text= (a regex matched against each
element’s collected subtree text), raced against BeautifulSoup.find_all(string=...); lxml, selectolax, and parsel
expose no equivalent, so this is a two-way race. When the text= filter is a plain string or a literal (no regex
metacharacters, case-sensitive) compiled pattern, turbohtml gathers each candidate’s collected text and matches it in C
– no Python str built, no per-element re.search call – where BeautifulSoup’s find_all(string=...) walks
the tree in Python, so turbohtml now leads on every page (it previously trailed on the larger ones). A case-insensitive
or otherwise non-literal pattern keeps the per-element Python path.
find |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
43 µs |
55.9 µs (1.3x) |
ars technica (56 kB) |
431 µs |
221 µs (0.6x) |
mozilla blog (95 kB) |
329 µs |
1.24 ms (3.8x) |
whatwg spec (235 kB) |
640 µs |
3.06 ms (4.8x) |
XPath 1.0 evaluation runs through xpath(), raced against lxml’s libxml2 engine, the XPath that
parsel, pyquery, and html5-parser all wrap (selectolax and BeautifulSoup have none). One expression per feature class
(name tests, the // abbreviation, attribute, positional, and arithmetic predicates, string and aggregate functions,
a reverse axis, a union, and a computed name test) runs over the 9.6 kB wpt page below; tox -e bench xpath repeats
the sweep across every page size. turbohtml compiles each expression against the tree once, resolves name tests to
interned atoms, and folds // to a single descendant walk, so it leads across the surface. The exception is a
predicate that references position() ([1] or position() <= 3): it pins the result to proximity order and
disables the // collapse, so on the largest pages lxml’s streaming evaluation closes the gap. The last eight rows
are the lxml/parsel options the parity work added: a $variable binding, an EXSLT re:test predicate (turbohtml’s
Python re against lxml’s C libexslt), an EXSLT set:distinct node-set reduction (built-in C dispatch on both
sides, so it races C against C), a smart_strings attribute read, a custom extensions= function, an
extensions= function whose return becomes a node-set feeding a later /@href step, a namespaces= prefix
binding that resolves //svg:rect against {"svg": ".../2000/svg"} over a page carrying an SVG block, and a
node-set $variable bound from a prior result ($rows/div, with rows reused from an earlier //div query)
fed into a later path step. turbohtml still leads, since lxml resolves the namespace map and option set on every call.
The last row precompiles the expression once with XPath and re-evaluates it, lxml’s etree.XPath
doing the same: both skip the per-call parse xpath() pays, and turbohtml’s compiled program stays
ahead per evaluation.
xpath (9.6 kB page) |
turbohtml |
|
|---|---|---|
|
1.7 µs |
11.5 µs (6.8x) |
|
400 ns |
3.9 µs (9.8x) |
|
1 µs |
10 µs (10.1x) |
|
800 ns |
6.3 µs (7.9x) |
|
9.51 µs |
10 µs (1.1x) |
|
400 ns |
4.2 µs (10.5x) |
|
5.4 µs |
13.6 µs (2.6x) |
|
400 ns |
2.4 µs (6.0x) |
|
600 ns |
3.2 µs (5.4x) |
|
5 µs |
14.7 µs (3.0x) |
|
400 ns |
2.4 µs (6.0x) |
|
500 ns |
4.3 µs (8.7x) |
|
400 ns |
6.2 µs (15.5x) |
|
400 ns |
4 µs (10.0x) |
|
500 ns |
2.8 µs (5.6x) |
|
900 ns |
3 µs (3.4x) |
|
950 ns |
3.5 µs (3.7x) |
|
600 ns |
3.2 µs (5.4x) |
|
2.5 µs |
5.4 µs (2.2x) |
|
300 ns |
3.1 µs (10.4x) |
Node paths¶
turbohtml.Element.css_path() and xpath_path() return the unique locator that re-finds an
element from the document root – a CSS selector and a positional XPath – against lxml’s getroottree().getpath(),
the libxml2 path builder devtools’ “copy selector” mirrors. lxml emits only the positional XPath, so getpath pairs
with both turbohtml methods. Each timed call walks every element in a pre-parsed page and serializes its path. Both
methods lead getpath by roughly five times across these pages, narrowing to under threefold on the spec.
css_path() previously rescanned the whole document to test each element’s id uniqueness, an
O(N2) cost over a page that made it slower than getpath on id-heavy pages; a cached per-tree id-occurrence
map (dropped with the element index on any mutation) now answers that test in O(1), so css_path keeps pace with the
positional xpath_path. The ratio on getpath is against xpath_path, the like-for-like locator.
input |
turbohtml css_path |
turbohtml xpath_path |
|
|---|---|---|---|
daring fireball (10 kB) |
18.9 µs |
18.9 µs |
104 µs (5.5x) |
ars technica (56 kB) |
91.4 µs |
98.8 µs |
557 µs (6.1x) |
mozilla blog (95 kB) |
227 µs |
277 µs |
1.43 ms (6.3x) |
whatwg spec (235 kB) |
2.24 ms |
2.2 ms |
6.11 ms (2.8x) |
Text content¶
The text suite collects the visible text two ways. First, the raw text join off a pre-parsed tree, the get_text
pass: turbohtml’s text property concatenates every descendant text run, against lxml’s
text_content(), selectolax’s text(), and BeautifulSoup’s get_text(). turbohtml gathers the runs in one C
walk into a buffer reserved up front, so it leads lxml by a small margin and selectolax and BeautifulSoup by roughly an
order of magnitude. parsel exposes no node-level text collector, so it sits out.
text content |
turbohtml |
|||
|---|---|---|---|---|
daring fireball (10 kB) |
2.7 µs |
3.4 µs (1.3x) |
19 µs (7.1x) |
22.5 µs (8.4x) |
ars technica (56 kB) |
13.4 µs |
16 µs (1.2x) |
76.5 µs (5.8x) |
91.7 µs (6.9x) |
mozilla blog (95 kB) |
23.3 µs |
25.8 µs (1.2x) |
166 µs (7.2x) |
205 µs (8.8x) |
whatwg spec (235 kB) |
76.8 µs |
89.5 µs (1.2x) |
588 µs (7.7x) |
765 µs (10.0x) |
Second, the layout-aware string-to-text extraction: turbohtml.Node.to_text() against inscriptis, the layout-aware HTML-to-text renderer it succeeds, html-text, Zyte’s plainer visible-text extractor, and resiliparse’s extract_plain_text. inscriptis and html-text both build an
lxml tree in Python and resiliparse renders text off the lexbor tree it parses to, where turbohtml does the whole layout
in one C walk; inscriptis additionally lays tables out as aligned columns, which html-text and resiliparse skip.
input |
turbohtml |
|||
|---|---|---|---|---|
article (2 KiB) |
2 µs |
20 µs (10.1x) |
101 µs (50.5x) |
180 µs (90.1x) |
table (4 KiB) |
10 µs |
54 µs (5.4x) |
270 µs (27.0x) |
910 µs (91.0x) |
collapsed (2 KiB) |
2 µs |
– |
106 µs (53.0x) |
– |
main (4 KiB) |
4 µs |
21 µs (5.3x) |
– |
– |
annotated (4 KiB) |
3 µs |
– |
– |
221 µs (73.7x) |
The collapsed row turns layout guessing off: turbohtml joins the stripped_strings word
stream against html-text’s extract_text(guess_layout=False); inscriptis and resiliparse have no comparable collapsed
mode. The main row strips page boilerplate first, main_text() against resiliparse’s
extract_plain_text(main_content=True). The annotated row labels matching elements with spans through
to_annotated_text() against inscriptis’s get_annotated_text; html-text and resiliparse have no
annotation surface, so they sit out that row.
Serializing¶
Serializing a parsed document back to HTML: turbohtml’s html, lxml’s tostring, selectolax’s
html, and BeautifulSoup’s decode. turbohtml scans each text run for the next character that needs escaping (two
code points at a time with the same SWAR lane probes escape() uses) and bulk-copies the clean spans,
recovering each special’s position from the lane mask, and reserves the whole-document buffer up front so the output
grows in one allocation. It serializes three to six times faster than lxml, over three times faster than selectolax, and
fifty to sixty times faster than BeautifulSoup.
serialize to HTML |
turbohtml |
|||
|---|---|---|---|---|
daring fireball (10 kB) |
7.1 µs |
28.6 µs (4.1x) |
39.9 µs (5.7x) |
430 µs (60.7x) |
ars technica (56 kB) |
40.7 µs |
157 µs (3.9x) |
198 µs (4.9x) |
1.9 ms (46.7x) |
mozilla blog (95 kB) |
79.2 µs |
312 µs (4.0x) |
432 µs (5.5x) |
4.17 ms (52.7x) |
whatwg spec (235 kB) |
209 µs |
750 µs (3.6x) |
835 µs (4.1x) |
11.1 ms (53.1x) |
Minifying¶
Minifying a document with turbohtml.clean.minify(): parse, then serialize once with every fold engaged (collapsing
insignificant whitespace, omitting the WHATWG-optional tags, unquoting attributes, and stripping comments), against
minify-html’s Rust minifier on the same folds (its CSS and JS
minification left off for a like-for-like comparison) and htmlmin’s pure-Python
HTMLParser walk. turbohtml parses and emits in C through one preallocated buffer, so with the parse included it runs
about twice as fast as minify-html and fourteen to twenty times faster than htmlmin.
minify a document |
turbohtml |
htmlmin |
|
|---|---|---|---|
daring fireball (10 kB) |
28.2 µs |
67.2 µs (2.4x) |
601 µs (21.4x) |
ars technica (56 kB) |
127 µs |
321 µs (2.6x) |
2.72 ms (21.6x) |
mozilla blog (95 kB) |
267 µs |
750 µs (2.9x) |
6.61 ms (24.8x) |
whatwg spec (235 kB) |
935 µs |
1.7 ms (1.9x) |
15.4 ms (16.5x) |
Building¶
The write path: construct a <ul> of N <li> rows from scratch (each with a class, a data attribute,
and a text child), then serialize it, the work an editor or template engine does. turbohtml’s arena allocation and
interned attribute names make construction cheaper than lxml’s libxml2 nodes and far cheaper than BeautifulSoup’s Python
objects. selectolax is parse-only, so it has no entry.
build a list |
turbohtml |
||
|---|---|---|---|
100 rows |
55.7 µs |
165 µs (3.0x) |
1.03 ms (18.5x) |
1000 rows |
570 µs |
1.38 ms (2.5x) |
8.6 ms (15.1x) |
10000 rows |
6.41 ms |
13.7 ms (2.2x) |
95 ms (14.9x) |
The construct and emit commands split that aggregate into its two halves over the same tree: construct
builds the rows and stops before serialization, and emit serializes a tree built once outside the timed region. The
split shows where each library spends the time – turbohtml’s arena keeps construction roughly twice as fast as lxml,
and its SWAR serializer pulls emit ahead by nearly six times, while BeautifulSoup pays its Python object cost on both
halves.
construct (no serialize) |
turbohtml |
||
|---|---|---|---|
100 rows |
43.4 µs |
100 µs (2.4x) |
265 µs (6.2x) |
1000 rows |
463 µs |
919 µs (2.0x) |
2.59 ms (5.6x) |
10000 rows |
4.66 ms |
9.7 ms (2.1x) |
25.4 ms (5.5x) |
emit a built tree |
turbohtml |
||
|---|---|---|---|
100 rows |
4.2 µs |
33.3 µs (8.0x) |
394 µs (93.9x) |
1000 rows |
42 µs |
325 µs (7.8x) |
4 ms (95.3x) |
10000 rows |
420 µs |
3.2 ms (7.7x) |
41.7 ms (99.3x) |
The terse turbohtml.build.E builder spells the same <ul> declaratively, raced against the dedicated HTML
generators dominate and yattag. E is two to three
times faster than dominate and on par with yattag – a touch behind it on the larger sweeps – and unlike either it
returns a real, queryable turbohtml tree rather than a string. That tree costs a little over twice the raw
Element constructor above – the price of the leading-mapping and per-child dispatch the sugar runs
in Python.
Editing¶
Editing a parsed tree: tag every <a> with rel="nofollow", a link-rewriting pass. Because the pass mutates the
tree, each library rebuilds a fresh parse before every iteration outside the timed region, then the timed call walks its
links and sets the attribute (turbohtml through the live attrs mapping, lxml through
Element.set, BeautifulSoup through item assignment). selectolax mutation is limited, so it has no entry.
tag every link |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
3 µs |
10.7 µs (3.6x) |
ars technica (56 kB) |
13 µs |
27.9 µs (2.2x) |
mozilla blog (95 kB) |
20 µs |
41.3 µs (2.1x) |
whatwg spec (235 kB) |
42 µs |
62 µs (1.5x) |
A second pass churns the class list: add then drop a token on every link (turbohtml’s
add_class()/remove_class() against lxml’s classes set). The
add-then-remove is a net no-op, so each repeat does equal work. BeautifulSoup has no class-token mutator, so it has no
entry.
class add/remove on every link |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
2.2 µs |
46 µs (21.0x) |
ars technica (56 kB) |
8.6 µs |
120 µs (14.0x) |
mozilla blog (95 kB) |
8.48 µs |
164 µs (19.4x) |
whatwg spec (235 kB) |
8.08 µs |
194 µs (24.1x) |
Two content setters replace the body’s children on a freshly parsed tree. set_inner_html()
reparses a fixed fragment in the <body>’s context and splices it in one C call, against lxml clearing the body and
appending fragments_fromstring, BeautifulSoup clearing it and appending a reparsed soup, and pyquery’s .html().
set_text() replaces the children with one verbatim text node, against pyquery’s .text().
replace body inner HTML |
turbohtml |
|||
|---|---|---|---|---|
daring fireball (10 kB) |
2.3 µs |
18 µs (7.9x) |
13.8 µs (6.0x) |
90.1 µs (39.2x) |
ars technica (56 kB) |
9.43 µs |
13.5 µs (1.5x) |
114 µs (12.1x) |
145 µs (15.4x) |
mozilla blog (95 kB) |
15.6 µs |
121 µs (7.8x) |
107 µs (6.9x) |
227 µs (14.6x) |
whatwg spec (235 kB) |
45.4 µs |
370 µs (8.2x) |
260 µs (5.8x) |
861 µs (19.0x) |
replace body text |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
1.2 µs |
32.8 µs (27.4x) |
ars technica (56 kB) |
7.7 µs |
12.9 µs (1.7x) |
mozilla blog (95 kB) |
13 µs |
124 µs (9.6x) |
whatwg spec (235 kB) |
39.8 µs |
371 µs (9.4x) |
A bulk tag edit over each page’s <code>/<a>/<q> elements: remove() drops each match
with its subtree, and strip_tags() unwraps each match but keeps its content. Both rewrites are
destructive, so the timed call parses the page afresh – the string-to-result transform these helpers perform – and
races each library’s own bulk tag helper. remove pairs with selectolax’s strip_tags (which drops matches with
their content) and pyquery’s .remove(); strip_tags pairs with w3lib’s regex remove_tags (which keeps the
content) and pyquery’s unwrap. turbohtml’s single C pass leads by roughly two to six times, the margin widest on the
smaller pages where the competitors’ per-match Python work has the least bulk text to hide behind.
|
turbohtml |
||
|---|---|---|---|
daring fireball (10 kB) |
25.3 µs |
166 µs (6.6x) |
107 µs (4.3x) |
ars technica (56 kB) |
125 µs |
349 µs (2.8x) |
499 µs (4.0x) |
mozilla blog (95 kB) |
283 µs |
1.2 ms (4.3x) |
1.61 ms (5.7x) |
whatwg spec (235 kB) |
687 µs |
3 ms (4.4x) |
2.9 ms (4.3x) |
Links¶
The link surface: extract every in-document link, resolve them against a base URL, and rewrite them through a callback.
turbohtml’s links(), resolve_links(), and
rewrite_links() against lxml.html’s iterlinks(), make_links_absolute(), and
rewrite_links() – the only other library that walks the link-bearing attributes (href, src, srcset,
…) as a set. Each operation runs over the three real saved pages and the 235 kB WHATWG spec; extraction is read-only
and rewrite applies an identity callback, so both reuse one cached parse, while absolutize rebuilds a fresh tree before
each iteration since make_links_absolute rewrites the hrefs in place. turbohtml walks the attribute set in C where
lxml re-resolves each URL in Python, so it leads from low single digits up to over a hundred times, the gap widening
with the link count on the page.
extract ( |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
9 µs |
145 µs (16.2x) |
ars technica (56 kB) |
27.7 µs |
538 µs (19.5x) |
mozilla blog (95 kB) |
51.5 µs |
1.2 ms (23.4x) |
whatwg spec (235 kB) |
84.8 µs |
3.9 ms (46.0x) |
absolutize ( |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
82 µs |
280 µs (3.5x) |
ars technica (56 kB) |
220 µs |
918 µs (4.2x) |
mozilla blog (95 kB) |
428 µs |
1.9 ms (4.5x) |
whatwg spec (235 kB) |
282 µs |
5.58 ms (19.8x) |
rewrite ( |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
3.2 µs |
163 µs (51.0x) |
ars technica (56 kB) |
13.4 µs |
605 µs (45.2x) |
mozilla blog (95 kB) |
23.1 µs |
1.38 ms (59.8x) |
whatwg spec (235 kB) |
41.9 µs |
4.4 ms (106x) |
Extraction¶
Pulling values out of a document, the idioms the parsel, pyquery, and w3lib migrations center on. First, reading every
matched node’s @href and visible text off a pre-parsed page: parsel’s ::attr/::text getall and a pyquery
.items() read against turbohtml selecting once and reading attr() and
text off each node. turbohtml compiles the selector once and reads interned atoms, where parsel
re-translates the CSS to XPath on libxml2 per call and pyquery boxes every match in a wrapper object, so it leads by
twenty-five to nearly a hundred times.
extract |
turbohtml |
||
|---|---|---|---|
daring fireball (10 kB) |
2.6 µs |
68 µs (26.2x) |
213 µs (81.9x) |
ars technica (56 kB) |
7 µs |
148 µs (21.2x) |
11 µs (1.6x) |
mozilla blog (95 kB) |
9.2 µs |
313 µs (34.1x) |
548 µs (59.6x) |
whatwg spec (235 kB) |
9.8 µs |
356 µs (36.4x) |
664 µs (67.8x) |
extract text (per match) |
turbohtml |
||
|---|---|---|---|
daring fireball (10 kB) |
2.7 µs |
97 µs (36.0x) |
117 µs (43.4x) |
ars technica (56 kB) |
6.7 µs |
10 µs (1.5x) |
292 µs (43.6x) |
mozilla blog (95 kB) |
10 µs |
351 µs (35.1x) |
471 µs (47.1x) |
whatwg spec (235 kB) |
10.9 µs |
380 µs (34.9x) |
260 µs (23.9x) |
Second, reading a document’s own URL hints: w3lib’s get_base_url and get_meta_refresh against turbohtml’s
base_url() and meta_refresh(). Both parse the string each call;
w3lib runs a regular-expression pass while turbohtml runs the WHATWG tree builder and reads the hint off the parsed
<head>, so the tree builder still comes out ahead of the regex on this small document.
url hint |
turbohtml |
|
|---|---|---|
|
1.2 µs |
7.4 µs (6.2x) |
|
1 µs |
6.5 µs (6.5x) |
Fluent chaining¶
A pyquery-style fluent chain over a pre-parsed tree: select every <a>, keep the linked ones, take the first, tag it,
and read its href (turbohtml’s turbohtml.query.Query against pyquery,
whose wrapper delegates to lxml). Both wrappers are thin Python over the underlying engine, so the gap is the engine’s:
turbohtml’s selector and attribute primitives run in C, and the wrapper avoids a redundant de-duplication when the chain
starts from one node, so it runs several to twenty times faster, depending on how much the page exercises the selector.
select, filter, tag, read |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
3.7 µs |
83.8 µs (22.7x) |
ars technica (56 kB) |
11.2 µs |
22.5 µs (2.1x) |
mozilla blog (95 kB) |
17 µs |
242 µs (14.3x) |
whatwg spec (235 kB) |
28.9 µs |
304 µs (10.6x) |
html.parser adapter¶
turbohtml.migration.stdlib.HTMLParser against the standard library’s html.parser.HTMLParser,
both subclassed with the same minimal handler so the comparison is the parser and dispatch cost for the identical
callback-driven programming model. The per-tag Python handler call is a floor both pay, so the margin is narrower than
raw tokenization, but turbohtml’s C tokenizer feeding the dispatch still runs it three to four times faster.
feed and dispatch a page |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
85.9 µs |
312 µs (3.7x) |
ars technica (56 kB) |
372 µs |
1.41 ms (3.8x) |
mozilla blog (95 kB) |
810 µs |
3.2 ms (4.0x) |
whatwg spec (235 kB) |
2.44 ms |
7.19 ms (3.0x) |
CSS minification¶
turbohtml.clean.minify_css() against the CSS minifiers on PyPI, over the unminified source CSS these frameworks
publish. Each minifier column pairs its output size with the time to produce it; the ratio in each cell is against
turbohtml. Sizes are deterministic byte counts; times are the minimum of repeated runs.
stylesheet |
turbohtml |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
size |
time |
size |
time |
size |
time |
size |
time |
size |
time |
size |
time |
|
normalize.css (6 kB) |
0 B |
44.6 µs |
0 B (0.06x) |
1.11 ms (24.9x) |
0 B (6.66x) |
466 µs (10.5x) |
1.8 kB (20799457.99x) |
1.5 ms (33.7x) |
1.8 kB (20855916.89x) |
525 µs (11.8x) |
1.8 kB (20709123.76x) |
624 µs (14.0x) |
animate.css (93 kB) |
0 B |
1.16 ms |
0 B (0.25x) |
25 ms (21.5x) |
0 B (10.75x) |
9.7 ms (8.4x) |
75.7 kB (110775036.20x) |
33.4 ms (28.8x) |
75.8 kB (110861330.10x) |
10.1 ms (8.7x) |
75.8 kB (110881806.61x) |
13.2 ms (11.4x) |
pico.css (90 kB) |
0 B |
1.43 ms |
0 B (0.14x) |
35.5 ms (24.8x) |
0 B (152.58x) |
225 ms (157x) |
81.6 kB (55871842.27x) |
47.4 ms (33.1x) |
81.8 kB (55992332.44x) |
290 ms (203x) |
81.9 kB (56054631.34x) |
295 ms (206x) |
foundation.css (164 kB) |
0 B |
3.05 ms |
0 B (0.17x) |
59.7 ms (19.7x) |
1 B (223.76x) |
505 ms (166x) |
136.4 kB (60768624.17x) |
79.6 ms (26.2x) |
136.3 kB (60741892.51x) |
659 ms (217x) |
136.5 kB (60812285.87x) |
664 ms (218x) |
bootstrap.css (274 kB) |
0 B |
4.14 ms |
0 B (0.09x) |
81.9 ms (19.8x) |
1 B (89.49x) |
598 ms (145x) |
234.2 kB (35371204.10x) |
110 ms (26.5x) |
232.4 kB (35092708.52x) |
776 ms (188x) |
234.3 kB (35385400.73x) |
786 ms (190x) |
bulma.css (745 kB) |
0 B |
11.2 ms |
0 B (0.09x) |
580 ms (52.0x) |
3 B (161.01x) |
2.78 s (250x) |
681.3 kB (35685269.08x) |
770 ms (69.0x) |
679.3 kB (35580568.29x) |
3.67 s (329x) |
681.3 kB (35685635.72x) |
3.71 s (332x) |
csscompressor (the YUI port) and cssmin (its BSD descendant) rewrite values to their shortest form the way
turbohtml does, but as pure-Python regex passes they turn quadratic on a large stylesheet and trail the C engine by 40x
to 800x. rcssmin is a C extension and faster than turbohtml, though it only strips comments and whitespace, so it
leaves a larger result everywhere except the custom-property-heavy bulma.css. css-html-js-minify is the slowest
of the set. The three pure-Python tools and rcssmin also break value safety: each rewrites the internal whitespace of a
custom-property value, which CSS Variables 1 §2 keeps as
the literal token stream that var() splices verbatim and getPropertyValue() reads back byte-exact, and
cssmin and css-html-js-minify collapse whitespace inside strings, so their output can change the cascade where
turbohtml’s round-trips. That rewrite is also the only reason rcssmin and cssmin end 0.1% ahead on
bulma.css, whose declarations are almost entirely custom properties.
lightningcss, the Rust binding, is a cascade-aware optimizer: it drops
declarations overridden elsewhere in the sheet and rewrites syntax for a browser-target set, so it reaches a smaller
size than turbohtml on most of the corpus (turbohtml comes out ahead on normalize.css). That target-dependent
optimization is the same idea as turbohtml’s baseline option carried further, and it is in scope. The cost shows in
the time half of each cell: it runs about 3x slower than turbohtml, and it rejects foundation.css with a parse error
on a media query the WHATWG recovery rules accept, where turbohtml minifies all six. turbohtml gives the smallest output
that stays value-safe at the most compatible baseline and recovers from malformed input.
JavaScript minification¶
turbohtml.clean.minify_js() against the PyPI JavaScript minifiers it replaces – rjsmin (a regex substitution), jsmin (Crockford’s
character state machine), and calmjs.parse (a full ES5 parser with an
obfuscating printer) – plus terser, the JavaScript ecosystem’s reference minifier, run
in-process under Node as the size bar. The inputs are real un-minified libraries, a size ladder every tool parses.
turbohtml renames every local binding (function and class declarations included) and runs the structural folds, so it
beats calmjs.parse’s heavier global obfuscation on size everywhere while running forty to eighty times faster, and it
lands within one percent of terser’s size at thirteen to twenty-five times less time; it trails only rjsmin on time,
which buys its speed by doing far less and leaving output roughly twice the size. Each cell pairs a minifier’s output
size with the time to produce it; both ratios are against turbohtml.
input |
turbohtml |
|||||||
|---|---|---|---|---|---|---|---|---|
size |
time |
size |
time |
size |
time |
size |
time |
|
underscore 1.13 (67 kB) |
0 B |
101 ms |
0 B (2.83x) |
80.5 µs (0.1x) |
34.0 kB (17147613.76x) |
111 µs (0.1x) |
34.0 kB (17148118.25x) |
8.33 ms (0.1x) |
backbone 1.6 (79 kB) |
0 B |
116 ms |
0 B (5.32x) |
69.5 µs (0.1x) |
35.2 kB (31038800.71x) |
94.9 µs (0.1x) |
35.2 kB (31039682.54x) |
8.75 ms (0.1x) |
jquery 3.7 (279 kB) |
0 B |
471 ms |
0 B (2.37x) |
372 µs (0.1x) |
141.1 kB (14166951.08x) |
481 µs (0.1x) |
141.4 kB (14192848.97x) |
35.4 ms (0.1x) |
lodash 4.17 (531 kB) |
0 B |
448 ms |
0 B (3.50x) |
574 µs (0.1x) |
148.8 kB (15901719.95x) |
796 µs (0.1x) |
148.8 kB (15902147.54x) |
48.3 ms (0.2x) |
Encoding detection¶
turbohtml.detect.detect() against the encoding detectors it replaces: chardet
(the pure-Python prober ensemble), charset-normalizer (decode-and-score,
what requests uses), and faust-cchardet (the maintained C binding
of uchardet; the original cchardet stops compiling at Python 3.11). turbohtml resolves certain input – a byte-order
mark, a <meta> declaration, valid UTF-8, pure ASCII – structurally before any scoring, which is where the 90x-2000x
rows come from, and its chardetng frequency scoring keeps declaration-less single-byte text about 3x ahead of chardet.
The one workload it loses is CJK-heavy bytes, where each CJK candidate drives a CPython incremental codec and uchardet’s
native tables stay ahead.
detect a byte stream’s encoding |
turbohtml |
|||
|---|---|---|---|---|
ascii (1 kB) |
1.89 µs |
912 ns (0.5x) |
46.1 µs (24.5x) |
118 µs (62.6x) |
utf-8 russian (4 kB) |
3.64 µs |
4.27 µs (1.2x) |
335 µs (91.9x) |
150 µs (41.3x) |
windows-1251 russian (4 kB) |
185 µs |
311 µs (1.7x) |
291 µs (1.6x) |
536 µs (2.9x) |
windows-1252 french (4 kB) |
186 µs |
359 µs (2.0x) |
978 µs (5.3x) |
674 µs (3.7x) |
shift_jis japanese (4 kB) |
708 µs |
33.7 µs (0.1x) |
326 µs (0.5x) |
547 µs (0.8x) |
utf-8 page (95 kB) |
642 ns |
52.8 µs (82.3x) |
565 µs (880x) |
1.27 ms (1979x) |
URL cleaning & link extraction¶
turbohtml.extract.clean_url(), normalize_url(), and
extract_links() against courlan, trafilatura’s URL
cleaner, and w3lib’s safe_url_string/canonicalize_url, Scrapy’s URL
utilities. The per-URL pass wins 2x-6x by scanning each component once in C-backed regexes and percent-encoding only
when a scan finds something to encode, where both competitors re-encode unconditionally through urllib’s per-character
quoters. Page-level extraction parses the real WHATWG DOM yet still finishes 1.4x-2.4x ahead of courlan’s regex scan,
because each distinct href is cleaned once and absolute links skip resolution.
clean and normalize 100 URLs |
turbohtml |
||
|---|---|---|---|
clean 100 URLs |
262 µs |
526 µs (2.1x) |
628 µs (2.4x) |
normalize 100 URLs |
216 µs |
404 µs (1.9x) |
1.27 ms (5.9x) |
extract filtered page links |
turbohtml |
|
|---|---|---|
daring fireball (10 kB) |
117 µs |
364 µs (3.2x) |
ars technica (56 kB) |
280 µs |
747 µs (2.7x) |
mozilla blog (95 kB) |
471 µs |
989 µs (2.1x) |
whatwg spec (235 kB) |
786 µs |
1.48 ms (1.9x) |