From pyquery¶
pyquery puts a jQuery-style fluent, chainable wrapper over lxml/cssselect, so you select and mutate a document with method
chains. A single PyQuery object holds a matched set of nodes; calling it with a CSS selector, chaining .filter,
.find, .eq, or .closest, and reading or writing attributes, text, HTML, and classes all return either a new
wrapper or a scalar. It is a common pick for scraping and templating code that wants the DOM-manipulation feel of jQuery
without a browser.
turbohtml covers that ground with turbohtml.query.Query, a fully type-annotated fluent wrapper whose selector,
attribute, and class primitives run in the C extension over a native tree, so the same chaining idiom ports across with
almost no rename and a large speed margin.
turbohtml vs pyquery¶
Dimension |
turbohtml |
pyquery |
|---|---|---|
Scope |
WHATWG HTML parser plus a jQuery-style |
jQuery-style wrapper only; parsing and the tree come from lxml |
Feature breadth |
CSS select/filter/traverse, attr/text/html/class ops, node-level mutation and XPath 1.0 |
CSS and XPath on the wrapper itself, broad jQuery method set, a network-fetching constructor |
Performance |
Selector and attribute primitives in C over a native tree; several to hundreds of times faster on the shared surface |
Python wrapper delegating to lxml/cssselect |
Typing |
Fully type annotated with shipped stubs |
No bundled type hints |
Dependencies |
Self-contained C extension, no Python runtime deps |
Depends on lxml and cssselect |
Maintenance |
Actively developed alongside the parser |
Mature, lightly maintained wrapper |
Feature overlap¶
The shared surface ports one-to-one from a matched set built by calling the wrapper with a selector:
Construction and selection:
PyQuery(html)->Query(parse(html)), thenquery("div.foo")andquery("a").find("span").Set narrowing and traversal:
filter(),eq(),closest(),parent(),children(),siblings()keep the same names.Reads and attribute writes:
attr(),text(),html()keep the same names.Class ops:
add_class(),remove_class(),toggle_class(),has_class(), available on bothQueryandElement.Iteration:
for item in pq("a").items()->for item in query("a").items().
What turbohtml adds¶
A full WHATWG HTML parser in the same package, so parsing and querying share one native tree instead of delegating to lxml.
C-resident selector and attribute primitives, giving the large speed margin measured below.
Shipped type stubs for the whole surface, including
Queryand the node API.No third-party runtime dependencies: the tree, selectors, and mutation all live in one self-contained C extension.
What pyquery has that turbohtml does not¶
A network-fetching constructor.
PyQuery(url=...)fetches over HTTP for you. turbohtml has no equivalent; fetch with httpx (or any client) and hand the bytes toturbohtml.parse().XPath on the fluent wrapper. pyquery exposes lxml’s
.xpath(...)directly on a matched set. turbohtml’sQueryis CSS-only; drop to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.``.wrap_all`` over a non-contiguous set. pyquery wraps any matched set in one container. turbohtml’s node API covers the tree-clean shapes (see below) but has no counterpart for an arbitrary scattered set;
append()the nodes into a new element and place it yourself.
Performance¶
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.21 µs |
6.96 µs (5.8x) |
parse to a tree — wpt small (4 kB) |
9.58 µs |
31.3 µs (3.3x) |
parse to a tree — wpt medium (9.6 kB) |
24.1 µs |
77.9 µs (3.3x) |
parse to a tree — wpt large (92 kB) |
209 µs |
877 µs (4.2x) |
parse to a tree — wpt CJK (124 kB) |
408 µs |
1.44 ms (3.6x) |
parse to a tree — whatwg spec (235 kB) |
400 µs |
1.66 ms (4.2x) |
find every anchor — daring fireball (10 kB) |
371 ns |
10.2 µs (27.4x) |
find every anchor — ars technica (56 kB) |
817 ns |
9.8 µs (12.0x) |
find every anchor — mozilla blog (95 kB) |
1.16 µs |
31.8 µs (27.4x) |
find every anchor — whatwg spec (235 kB) |
1.36 µs |
60.1 µs (44.3x) |
select div a[href] — daring fireball (10 kB) |
610 ns |
29.6 µs (48.6x) |
select div a[href] — ars technica (56 kB) |
1.47 µs |
17.2 µs (11.8x) |
select div a[href] — mozilla blog (95 kB) |
2.1 µs |
810 µs (386x) |
select div a[href] — whatwg spec (235 kB) |
1.78 µs |
1.42 ms (796x) |
match each anchor against div a[href] — daring fireball (10 kB) |
1.79 µs |
432 µs (242x) |
match each anchor against div a[href] — ars technica (56 kB) |
4.21 µs |
10.4 µs (2.5x) |
match each anchor against div a[href] — mozilla blog (95 kB) |
5.76 µs |
1.41 ms (245x) |
match each anchor against div a[href] — whatwg spec (235 kB) |
6.68 µs |
1.69 ms (253x) |
find by text content — daring fireball (10 kB) |
24.4 µs |
51.4 µs (2.2x) |
find by text content — ars technica (56 kB) |
178 µs |
239 µs (1.4x) |
find by text content — mozilla blog (95 kB) |
287 µs |
440 µs (1.6x) |
find by text content — whatwg spec (235 kB) |
560 µs |
1.18 ms (2.2x) |
collect visible text — daring fireball (10 kB) |
2.7 µs |
128 µs (47.4x) |
collect visible text — ars technica (56 kB) |
13 µs |
10 µs (0.8x) |
collect visible text — mozilla blog (95 kB) |
22.4 µs |
913 µs (40.9x) |
collect visible text — whatwg spec (235 kB) |
75.4 µs |
4.85 ms (64.4x) |
serialize a parsed tree — daring fireball (10 kB) |
7.02 µs |
16.9 µs (2.5x) |
serialize a parsed tree — ars technica (56 kB) |
38.8 µs |
75.3 µs (2.0x) |
serialize a parsed tree — mozilla blog (95 kB) |
76.3 µs |
159 µs (2.1x) |
serialize a parsed tree — whatwg spec (235 kB) |
195 µs |
379 µs (2.0x) |
tag every link rel=nofollow — daring fireball (10 kB) |
3.32 µs |
19 µs (5.8x) |
tag every link rel=nofollow — ars technica (56 kB) |
13.1 µs |
56.3 µs (4.4x) |
tag every link rel=nofollow — mozilla blog (95 kB) |
20.9 µs |
57.2 µs (2.8x) |
tag every link rel=nofollow — whatwg spec (235 kB) |
42.6 µs |
308 µs (7.3x) |
class add/remove on every link — daring fireball (10 kB) |
2.24 µs |
43.5 µs (19.4x) |
class add/remove on every link — ars technica (56 kB) |
8.89 µs |
10 µs (1.2x) |
class add/remove on every link — mozilla blog (95 kB) |
8.78 µs |
151 µs (17.2x) |
class add/remove on every link — whatwg spec (235 kB) |
8.7 µs |
187 µs (21.5x) |
drop tags with content (remove) — daring fireball (10 kB) |
23.8 µs |
155 µs (6.5x) |
drop tags with content (remove) — ars technica (56 kB) |
115 µs |
323 µs (2.8x) |
drop tags with content (remove) — mozilla blog (95 kB) |
260 µs |
1.12 ms (4.4x) |
drop tags with content (remove) — whatwg spec (235 kB) |
645 µs |
2.62 ms (4.1x) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
24.3 µs |
175 µs (7.3x) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
122 µs |
323 µs (2.7x) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
264 µs |
1.18 ms (4.5x) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
683 µs |
3.06 ms (4.5x) |
replace body inner HTML — daring fireball (10 kB) |
2.12 µs |
18.1 µs (8.6x) |
replace body inner HTML — ars technica (56 kB) |
8.69 µs |
13.3 µs (1.6x) |
replace body inner HTML — mozilla blog (95 kB) |
14.4 µs |
123 µs (8.6x) |
replace body inner HTML — whatwg spec (235 kB) |
40.1 µs |
370 µs (9.3x) |
replace body text — daring fireball (10 kB) |
1.18 µs |
15.4 µs (13.1x) |
replace body text — ars technica (56 kB) |
7.73 µs |
13.9 µs (1.8x) |
replace body text — mozilla blog (95 kB) |
12.9 µs |
122 µs (9.5x) |
replace body text — whatwg spec (235 kB) |
40.6 µs |
365 µs (9.0x) |
walk every descendant — daring fireball (10 kB) |
3.26 µs |
76.7 µs (23.6x) |
walk every descendant — ars technica (56 kB) |
13.4 µs |
227 µs (17.0x) |
walk every descendant — mozilla blog (95 kB) |
28 µs |
619 µs (22.1x) |
walk every descendant — whatwg spec (235 kB) |
96.8 µs |
2.15 ms (22.2x) |
fluent jQuery-style chain — daring fireball (10 kB) |
3.69 µs |
87.5 µs (23.8x) |
fluent jQuery-style chain — ars technica (56 kB) |
11.3 µs |
22.8 µs (2.1x) |
fluent jQuery-style chain — mozilla blog (95 kB) |
16.9 µs |
245 µs (14.5x) |
fluent jQuery-style chain — whatwg spec (235 kB) |
29 µs |
313 µs (10.9x) |
extract every link — daring fireball (10 kB) |
8.49 µs |
167 µs (19.7x) |
extract every link — ars technica (56 kB) |
27.6 µs |
11.5 µs (0.5x) |
extract every link — mozilla blog (95 kB) |
50.1 µs |
598 µs (12.0x) |
extract every link — whatwg spec (235 kB) |
84.6 µs |
719 µs (8.5x) |
absolutize every link — daring fireball (10 kB) |
82.6 µs |
386 µs (4.7x) |
absolutize every link — ars technica (56 kB) |
212 µs |
13 µs (0.1x) |
absolutize every link — mozilla blog (95 kB) |
482 µs |
1.38 ms (2.9x) |
absolutize every link — whatwg spec (235 kB) |
270 µs |
1.69 ms (6.3x) |
rewrite every link — daring fireball (10 kB) |
4.69 µs |
299 µs (63.8x) |
rewrite every link — ars technica (56 kB) |
11.5 µs |
11.1 µs (1.0x) |
rewrite every link — mozilla blog (95 kB) |
21.7 µs |
999 µs (46.2x) |
rewrite every link — whatwg spec (235 kB) |
38.1 µs |
1.16 ms (30.4x) |
social-card extraction — head |
1.79 µs |
73 µs (40.7x) |
social-card extraction — article 8 KiB |
22.1 µs |
163 µs (7.4x) |
extract @href per match — daring fireball (10 kB) |
2.49 µs |
162 µs (65.1x) |
extract @href per match — ars technica (56 kB) |
6.29 µs |
10.5 µs (1.7x) |
extract @href per match — mozilla blog (95 kB) |
8.51 µs |
756 µs (88.9x) |
extract @href per match — whatwg spec (235 kB) |
9.05 µs |
649 µs (71.8x) |
extract text per match — daring fireball (10 kB) |
2.75 µs |
92 µs (33.5x) |
extract text per match — ars technica (56 kB) |
6.39 µs |
10.3 µs (1.7x) |
extract text per match — mozilla blog (95 kB) |
10.3 µs |
345 µs (33.7x) |
extract text per match — whatwg spec (235 kB) |
10.8 µs |
384 µs (35.5x) |
extract URL hints — base_url / get_base_url |
1.27 µs |
18.4 µs (14.6x) |
extract URL hints — meta_refresh / get_meta_refresh |
1.29 µs |
26.6 µs (20.7x) |
extract filtered page links — daring fireball (10 kB) |
127 µs |
167 µs (1.4x) |
extract filtered page links — ars technica (56 kB) |
310 µs |
10.5 µs (0.1x) |
extract filtered page links — mozilla blog (95 kB) |
514 µs |
541 µs (1.1x) |
extract filtered page links — whatwg spec (235 kB) |
889 µs |
631 µs (0.8x) |
The whole shared surface – chaining a select/filter/read, setting content, bulk-editing tags, and reading a value off every match – runs several to hundreds of times faster because the wrapper edits its native tree in C and skips a redundant de-duplication when a chain starts from a single node, where pyquery drives lxml under its jQuery-style wrapper.
How to migrate¶
Build a Query from a parsed document and call it with a selector; the method chains port
almost name for name:
from turbohtml import parse
from turbohtml.query import Query
query = Query(parse("<div><a href='/u'>l</a><a>m</a></div>"))
print(query("a").filter("[href]").eq(0).add_class("seen").attr("href"))
print([anchor.text() for anchor in query("a").items()])
/u
['l', 'm']
turbohtml |
|
|---|---|
|
|
|
|
|
the same names |
|
the same names |
|
the same names |
|
|
|
the same names |
iterating |
|
jQuery |
|
jQuery |
pyquery’s .wrap_all(html) wraps a whole matched set in one new container in place; the node API has two methods for
the shapes that fit a tree model cleanly. wrap_children() boxes every child of a container, and
wrap_siblings() wraps a node and the contiguous run of siblings after it (through an until
node, or to the last sibling), so query("p").wrap_all("<div/>") over a run of adjacent paragraphs becomes
first.wrap_siblings(Element("div"), until=last):
turbohtml |
|
|---|---|
|
|
|
pyquery’s content setters – .html(markup) reparses a matched element’s children and .text(s) replaces them with
one verbatim text node – map onto three element methods. set_inner_html() parses the markup as
a fragment in the element’s context and replaces its children; set_text() replaces them with
one verbatim text node; and insert_adjacent_html() splices a parsed fragment at a DOM position
(the .append(markup) / insertAdjacentHTML shape):
turbohtml |
|
|---|---|
|
|
|
|
|
Gotchas and pitfalls¶
.wrap_allover an arbitrary, non-contiguous set of nodes has no single node-method counterpart (the set has no shared anchor to place the wrapper at); wrap the contiguous run, orappend()the scattered nodes into one new element and place it yourself.pyquery’s network-fetching constructor (
PyQuery(url=...)) is out of scope: fetch with httpx (or any client) and hand the bytes toturbohtml.parse().pyquery exposes lxml’s
.xpath(...)on the fluent wrapper itself; turbohtml’sQueryis CSS-only, so an XPath chain drops to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.turbohtml parses to the WHATWG spec, so a malformed document is fixed up exactly as a browser would (implied
<tbody>, reparented<head>content); pyquery’s tree follows lxml’s HTML parser, which can differ on the same broken input.