From pyquery¶
pyquery puts a jQuery-style fluent, chainable wrapper over lxml/cssselect, so you select and mutate a document with method
chains. A single PyQuery object holds a matched set of nodes; calling it with a CSS selector, chaining .filter,
.find, .eq, or .closest, and reading or writing attributes, text, HTML, and classes all return either a new
wrapper or a scalar. It is a common pick for scraping and templating code that wants the DOM-manipulation feel of jQuery
without a browser.
turbohtml covers that ground with turbohtml.query.Query, a fully type-annotated fluent wrapper whose selector,
attribute, and class primitives run in the C extension over a native tree, so the same chaining idiom ports across with
almost no rename and a large speed margin.
turbohtml vs pyquery¶
Dimension |
turbohtml |
pyquery |
|---|---|---|
Scope |
WHATWG HTML parser plus a jQuery-style |
jQuery-style wrapper only; parsing and the tree come from lxml |
Feature breadth |
CSS select/filter/traverse, attr/text/html/class ops, node-level mutation and XPath 1.0 |
CSS and XPath on the wrapper itself, broad jQuery method set, a network-fetching constructor |
Performance |
Selector and attribute primitives in C over a native tree; several to ~100x faster on the shared surface |
Python wrapper delegating to lxml/cssselect |
Typing |
Fully type annotated with shipped stubs |
No bundled type hints |
Dependencies |
Self-contained C extension, no Python runtime deps |
Depends on lxml and cssselect |
Maintenance |
Actively developed alongside the parser |
Mature, lightly maintained wrapper |
Feature overlap¶
The shared surface ports one-to-one from a matched set built by calling the wrapper with a selector:
Construction and selection:
PyQuery(html)->Query(parse(html)), thenquery("div.foo")andquery("a").find("span").Set narrowing and traversal:
filter(),eq(),closest(),parent(),children(),siblings()keep the same names.Reads and attribute writes:
attr(),text(),html()keep the same names.Class ops:
add_class(),remove_class(),toggle_class(),has_class(), available on bothQueryandElement.Iteration:
for item in pq("a").items()->for item in query("a").items().
What turbohtml adds¶
A full WHATWG HTML parser in the same package, so parsing and querying share one native tree instead of delegating to lxml.
C-resident selector and attribute primitives, giving the large speed margin measured below.
Shipped type stubs for the whole surface, including
Queryand the node API.No third-party runtime dependencies: the tree, selectors, and mutation all live in one self-contained C extension.
What pyquery has that turbohtml does not¶
A network-fetching constructor.
PyQuery(url=...)fetches over HTTP for you. turbohtml has no equivalent; fetch with httpx (or any client) and hand the bytes toturbohtml.parse().XPath on the fluent wrapper. pyquery exposes lxml’s
.xpath(...)directly on a matched set. turbohtml’sQueryis CSS-only; drop to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.``.wrap_all`` over a non-contiguous set. pyquery wraps any matched set in one container. turbohtml’s node API covers the tree-clean shapes (see below) but has no counterpart for an arbitrary scattered set;
append()the nodes into a new element and place it yourself.
Performance¶
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.21 µs |
7.18 µs (6.0x) |
parse to a tree — wpt small (4 kB) |
9.57 µs |
31.5 µs (3.3x) |
parse to a tree — wpt medium (9.6 kB) |
24.2 µs |
81.4 µs (3.4x) |
parse to a tree — wpt large (92 kB) |
207 µs |
874 µs (4.3x) |
parse to a tree — wpt CJK (124 kB) |
407 µs |
1.43 ms (3.6x) |
parse to a tree — whatwg spec (235 kB) |
405 µs |
1.69 ms (4.2x) |
find every anchor — daring fireball (10 kB) |
375 ns |
9.97 µs (26.6x) |
find every anchor — ars technica (56 kB) |
828 ns |
10.1 µs (12.3x) |
find every anchor — mozilla blog (95 kB) |
1.18 µs |
31.8 µs (27.0x) |
find every anchor — whatwg spec (235 kB) |
1.32 µs |
61.9 µs (47.1x) |
select div a[href] — daring fireball (10 kB) |
639 ns |
29.5 µs (46.2x) |
select div a[href] — ars technica (56 kB) |
1.46 µs |
18.5 µs (12.7x) |
select div a[href] — mozilla blog (95 kB) |
2.07 µs |
807 µs (391x) |
select div a[href] — whatwg spec (235 kB) |
1.74 µs |
1.37 ms (786x) |
find by text content — daring fireball (10 kB) |
42.9 µs |
51.3 µs (1.2x) |
find by text content — ars technica (56 kB) |
431 µs |
256 µs (0.6x) |
find by text content — mozilla blog (95 kB) |
329 µs |
443 µs (1.4x) |
find by text content — whatwg spec (235 kB) |
638 µs |
1.4 ms (2.2x) |
collect visible text — daring fireball (10 kB) |
2.69 µs |
128 µs (47.7x) |
collect visible text — ars technica (56 kB) |
13.4 µs |
11.1 µs (0.9x) |
collect visible text — mozilla blog (95 kB) |
23.3 µs |
926 µs (39.8x) |
collect visible text — whatwg spec (235 kB) |
76.8 µs |
4.91 ms (63.9x) |
serialize a parsed tree — daring fireball (10 kB) |
7.14 µs |
17.1 µs (2.4x) |
serialize a parsed tree — ars technica (56 kB) |
40.7 µs |
76.2 µs (1.9x) |
serialize a parsed tree — mozilla blog (95 kB) |
79.2 µs |
156 µs (2.0x) |
serialize a parsed tree — whatwg spec (235 kB) |
209 µs |
382 µs (1.9x) |
tag every link rel=nofollow — daring fireball (10 kB) |
3.32 µs |
20.8 µs (6.3x) |
tag every link rel=nofollow — ars technica (56 kB) |
13.2 µs |
63.5 µs (4.9x) |
tag every link rel=nofollow — mozilla blog (95 kB) |
21 µs |
61.8 µs (3.0x) |
tag every link rel=nofollow — whatwg spec (235 kB) |
42.5 µs |
333 µs (7.9x) |
class add/remove on every link — daring fireball (10 kB) |
2.2 µs |
42.9 µs (19.6x) |
class add/remove on every link — ars technica (56 kB) |
8.55 µs |
10.2 µs (1.2x) |
class add/remove on every link — mozilla blog (95 kB) |
8.48 µs |
146 µs (17.2x) |
class add/remove on every link — whatwg spec (235 kB) |
8.08 µs |
184 µs (22.8x) |
drop tags with content (remove) — daring fireball (10 kB) |
25.3 µs |
166 µs (6.6x) |
drop tags with content (remove) — ars technica (56 kB) |
125 µs |
349 µs (2.8x) |
drop tags with content (remove) — mozilla blog (95 kB) |
283 µs |
1.24 ms (4.4x) |
drop tags with content (remove) — whatwg spec (235 kB) |
687 µs |
2.97 ms (4.4x) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
25.8 µs |
184 µs (7.2x) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
127 µs |
335 µs (2.7x) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
286 µs |
1.22 ms (4.3x) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
742 µs |
3.04 ms (4.2x) |
replace body inner HTML — daring fireball (10 kB) |
2.33 µs |
18 µs (7.8x) |
replace body inner HTML — ars technica (56 kB) |
9.43 µs |
13.5 µs (1.5x) |
replace body inner HTML — mozilla blog (95 kB) |
15.6 µs |
121 µs (7.8x) |
replace body inner HTML — whatwg spec (235 kB) |
45.4 µs |
369 µs (8.2x) |
replace body text — daring fireball (10 kB) |
1.19 µs |
32.8 µs (27.7x) |
replace body text — ars technica (56 kB) |
7.72 µs |
12.9 µs (1.7x) |
replace body text — mozilla blog (95 kB) |
13 µs |
124 µs (9.6x) |
replace body text — whatwg spec (235 kB) |
39.8 µs |
371 µs (9.4x) |
walk every descendant — daring fireball (10 kB) |
3.34 µs |
76.6 µs (23.0x) |
walk every descendant — ars technica (56 kB) |
13.8 µs |
243 µs (17.7x) |
walk every descendant — mozilla blog (95 kB) |
28.2 µs |
609 µs (21.6x) |
walk every descendant — whatwg spec (235 kB) |
101 µs |
2.14 ms (21.3x) |
fluent jQuery-style chain — daring fireball (10 kB) |
3.68 µs |
83.8 µs (22.8x) |
fluent jQuery-style chain — ars technica (56 kB) |
11.2 µs |
22.5 µs (2.1x) |
fluent jQuery-style chain — mozilla blog (95 kB) |
16.6 µs |
242 µs (14.6x) |
fluent jQuery-style chain — whatwg spec (235 kB) |
28.9 µs |
304 µs (10.5x) |
extract every link — daring fireball (10 kB) |
8.66 µs |
165 µs (19.1x) |
extract every link — ars technica (56 kB) |
27.7 µs |
10.4 µs (0.4x) |
extract every link — mozilla blog (95 kB) |
51.5 µs |
528 µs (10.3x) |
extract every link — whatwg spec (235 kB) |
84.8 µs |
674 µs (8.0x) |
absolutize every link — daring fireball (10 kB) |
82 µs |
453 µs (5.6x) |
absolutize every link — ars technica (56 kB) |
220 µs |
16.6 µs (0.1x) |
absolutize every link — mozilla blog (95 kB) |
428 µs |
1.51 ms (3.6x) |
absolutize every link — whatwg spec (235 kB) |
282 µs |
1.9 ms (6.8x) |
rewrite every link — daring fireball (10 kB) |
3.24 µs |
326 µs (101x) |
rewrite every link — ars technica (56 kB) |
13.4 µs |
10.5 µs (0.8x) |
rewrite every link — mozilla blog (95 kB) |
23.1 µs |
1.12 ms (48.5x) |
rewrite every link — whatwg spec (235 kB) |
41.9 µs |
1.27 ms (30.3x) |
social-card extraction — head |
1.9 µs |
76.7 µs (40.3x) |
social-card extraction — article 8 KiB |
25.3 µs |
169 µs (6.8x) |
extract @href per match — daring fireball (10 kB) |
2.58 µs |
213 µs (82.4x) |
extract @href per match — ars technica (56 kB) |
6.65 µs |
10.8 µs (1.7x) |
extract @href per match — mozilla blog (95 kB) |
9.23 µs |
548 µs (59.4x) |
extract @href per match — whatwg spec (235 kB) |
9.81 µs |
664 µs (67.7x) |
extract text per match — daring fireball (10 kB) |
2.69 µs |
97.4 µs (36.2x) |
extract text per match — ars technica (56 kB) |
6.68 µs |
10.3 µs (1.6x) |
extract text per match — mozilla blog (95 kB) |
10.2 µs |
351 µs (34.4x) |
extract text per match — whatwg spec (235 kB) |
10.9 µs |
380 µs (34.9x) |
extract URL hints — base_url / get_base_url |
1.25 µs |
18.4 µs (14.8x) |
extract URL hints — meta_refresh / get_meta_refresh |
1.31 µs |
26.3 µs (20.2x) |
extract filtered page links — daring fireball (10 kB) |
117 µs |
164 µs (1.5x) |
extract filtered page links — ars technica (56 kB) |
280 µs |
10.6 µs (0.1x) |
extract filtered page links — mozilla blog (95 kB) |
471 µs |
528 µs (1.2x) |
extract filtered page links — whatwg spec (235 kB) |
786 µs |
653 µs (0.9x) |
The whole shared surface – chaining a select/filter/read, setting content, bulk-editing tags, and reading a value off every match – runs several to a hundred times faster because the wrapper edits its native tree in C and skips a redundant de-duplication when a chain starts from a single node, where pyquery drives lxml under its jQuery-style wrapper.
How to migrate¶
Build a Query from a parsed document and call it with a selector; the method chains port
almost name for name:
from turbohtml import parse
from turbohtml.query import Query
query = Query(parse("<div><a href='/u'>l</a><a>m</a></div>"))
print(query("a").filter("[href]").eq(0).add_class("seen").attr("href"))
print([anchor.text() for anchor in query("a").items()])
/u
['l', 'm']
turbohtml |
|
|---|---|
|
|
|
|
|
the same names |
|
the same names |
|
the same names |
|
|
|
the same names |
iterating |
|
jQuery |
|
jQuery |
pyquery’s .wrap_all(html) wraps a whole matched set in one new container in place; the node API has two methods for
the shapes that fit a tree model cleanly. wrap_children() boxes every child of a container, and
wrap_siblings() wraps a node and the contiguous run of siblings after it (through an until
node, or to the last sibling), so query("p").wrap_all("<div/>") over a run of adjacent paragraphs becomes
first.wrap_siblings(Element("div"), until=last):
turbohtml |
|
|---|---|
|
|
|
pyquery’s content setters – .html(markup) reparses a matched element’s children and .text(s) replaces them with
one verbatim text node – map onto three element methods. set_inner_html() parses the markup as
a fragment in the element’s context and replaces its children; set_text() replaces them with
one verbatim text node; and insert_adjacent_html() splices a parsed fragment at a DOM position
(the .append(markup) / insertAdjacentHTML shape):
turbohtml |
|
|---|---|
|
|
|
|
|
Gotchas and pitfalls¶
.wrap_allover an arbitrary, non-contiguous set of nodes has no single node-method counterpart (the set has no shared anchor to place the wrapper at); wrap the contiguous run, orappend()the scattered nodes into one new element and place it yourself.pyquery’s network-fetching constructor (
PyQuery(url=...)) is out of scope: fetch with httpx (or any client) and hand the bytes toturbohtml.parse().pyquery exposes lxml’s
.xpath(...)on the fluent wrapper itself; turbohtml’sQueryis CSS-only, so an XPath chain drops to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.turbohtml parses to the WHATWG spec, so a malformed document is fixed up exactly as a browser would (implied
<tbody>, reparented<head>content); pyquery’s tree follows lxml’s HTML parser, which can differ on the same broken input.