From pyquery¶
pyquery puts a jQuery-style fluent, chainable wrapper over lxml/cssselect, so you select and mutate a document with method
chains. A single PyQuery object holds a matched set of nodes; calling it with a CSS selector, chaining .filter,
.find, .eq, or .closest, and reading or writing attributes, text, HTML, and classes all return either a new
wrapper or a scalar. It is a common pick for scraping and templating code that wants the DOM-manipulation feel of jQuery
without a browser.
turbohtml covers that ground with turbohtml.query.Query, a fully type-annotated fluent wrapper whose selector,
attribute, and class primitives run in the C extension over a native tree, so the same chaining idiom ports across with
almost no rename and a large speed margin.
turbohtml vs pyquery¶
Dimension |
turbohtml |
pyquery |
|---|---|---|
Scope |
WHATWG HTML parser plus a jQuery-style |
jQuery-style wrapper only; parsing and the tree come from lxml |
Feature breadth |
CSS select/filter/traverse, attr/text/html/class ops, node-level mutation and XPath 1.0 |
CSS and XPath on the wrapper itself, broad jQuery method set, a network-fetching constructor |
Performance |
Selector and attribute primitives in C over a native tree; several to hundreds of times faster on the shared surface |
Python wrapper delegating to lxml/cssselect |
Typing |
Fully type annotated with shipped stubs |
No bundled type hints |
Dependencies |
Self-contained C extension, no Python runtime deps |
Depends on lxml and cssselect |
Maintenance |
Actively developed alongside the parser |
Mature, lightly maintained wrapper |
Feature overlap¶
The shared surface ports one-to-one from a matched set built by calling the wrapper with a selector:
Construction and selection:
PyQuery(html)->Query(parse(html)), thenquery("div.foo")andquery("a").find("span").Set narrowing and traversal:
filter(),eq(),closest(),parent(),children(),siblings()keep the same names.Reads and attribute writes:
attr(),text(),html()keep the same names.Class ops:
add_class(),remove_class(),toggle_class(),has_class(), available on bothQueryandElement.Iteration:
for item in pq("a").items()->for item in query("a").items().
What turbohtml adds¶
A full WHATWG HTML parser in the same package, so parsing and querying share one native tree instead of delegating to lxml.
C-resident selector and attribute primitives, giving the large speed margin measured below.
Shipped type stubs for the whole surface, including
Queryand the node API.No third-party runtime dependencies: the tree, selectors, and mutation all live in one self-contained C extension.
What pyquery has that turbohtml does not¶
A network-fetching constructor.
PyQuery(url=...)fetches over HTTP for you. turbohtml has no equivalent; fetch with httpx (or any client) and hand the bytes toturbohtml.parse().XPath on the fluent wrapper. pyquery exposes lxml’s
.xpath(...)directly on a matched set. turbohtml’sQueryis CSS-only; drop to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.``.wrap_all`` over a non-contiguous set. pyquery wraps any matched set in one container. turbohtml’s node API covers the tree-clean shapes (see below) but has no counterpart for an arbitrary scattered set;
append()the nodes into a new element and place it yourself.
Performance¶
operation |
turbohtml |
|
|---|---|---|
parse to a tree — wpt tiny (0.6 kB) |
1.53 µs |
6.19 µs (4.1x ±3%) |
parse to a tree — wpt small (4 kB) |
12.2 µs |
38.4 µs (3.2x ±4%) |
parse to a tree — wpt medium (9.6 kB) |
32.5 µs |
101 µs (3.2x ±7%) |
parse to a tree — wpt large (92 kB) |
276 µs |
884 µs (3.2x ±3%) |
parse to a tree — wpt CJK (124 kB) |
532 µs |
2.02 ms (3.9x ±7%) |
parse to a tree — whatwg spec (235 kB) |
519 µs |
1.68 ms (3.3x ±3%) |
parse to a tree — common tags (13 kB) |
78.5 µs |
225 µs (2.9x ±3%) |
find every anchor — daring fireball (10 kB) |
372 ns |
9.77 µs (26.3x ±1%) |
find every anchor — ars technica (56 kB) |
811 ns |
20 µs (24.8x ±2%) |
find every anchor — mozilla blog (95 kB) |
1.12 µs |
29.5 µs (26.4x ±2%) |
find every anchor — whatwg spec (235 kB) |
1.3 µs |
60 µs (46.3x ±2%) |
select div a[href] — daring fireball (10 kB) |
595 ns |
29.8 µs (50.1x ±2%) |
select div a[href] — ars technica (56 kB) |
1.43 µs |
127 µs (88.5x ±1%) |
select div a[href] — mozilla blog (95 kB) |
2.01 µs |
801 µs (400x ±1%) |
select div a[href] — whatwg spec (235 kB) |
1.76 µs |
1.42 ms (804x ±6%) |
match each anchor against div a[href] — daring fireball (10 kB) |
1.85 µs |
431 µs (234x ±3%) |
match each anchor against div a[href] — ars technica (56 kB) |
4.23 µs |
1.02 ms (242x ±2%) |
match each anchor against div a[href] — mozilla blog (95 kB) |
5.83 µs |
1.41 ms (243x ±1%) |
match each anchor against div a[href] — whatwg spec (235 kB) |
6.46 µs |
1.66 ms (258x ±2%) |
find by text content — daring fireball (10 kB) |
24.4 µs |
50.6 µs (2.1x ±2%) |
find by text content — ars technica (56 kB) |
173 µs |
236 µs (1.4x ±1%) |
find by text content — mozilla blog (95 kB) |
278 µs |
441 µs (1.6x ±1%) |
find by text content — whatwg spec (235 kB) |
556 µs |
1.17 ms (2.1x ±1%) |
collect visible text — daring fireball (10 kB) |
2.63 µs |
127 µs (48.4x ±1%) |
collect visible text — ars technica (56 kB) |
13.5 µs |
448 µs (33.2x ±1%) |
collect visible text — mozilla blog (95 kB) |
21.9 µs |
906 µs (41.5x ±1%) |
collect visible text — whatwg spec (235 kB) |
83.7 µs |
4.8 ms (57.4x ±1%) |
serialize a parsed tree — daring fireball (10 kB) |
6.52 µs |
17.2 µs (2.7x ±2%) |
serialize a parsed tree — ars technica (56 kB) |
34.8 µs |
76.7 µs (2.3x ±2%) |
serialize a parsed tree — mozilla blog (95 kB) |
67.4 µs |
155 µs (2.3x ±3%) |
serialize a parsed tree — whatwg spec (235 kB) |
174 µs |
377 µs (2.2x ±2%) |
tag every link rel=nofollow — daring fireball (10 kB) |
4.74 µs |
37.9 µs (8.1x ±8%) |
tag every link rel=nofollow — ars technica (56 kB) |
16.6 µs |
122 µs (7.4x ±25%) |
tag every link rel=nofollow — mozilla blog (95 kB) |
25.4 µs |
221 µs (8.8x ±16%) |
tag every link rel=nofollow — whatwg spec (235 kB) |
51.4 µs |
388 µs (7.6x ±6%) |
class add/remove on every link — daring fireball (10 kB) |
2.62 µs |
41.5 µs (15.9x ±2%) |
class add/remove on every link — ars technica (56 kB) |
9.67 µs |
108 µs (11.2x ±2%) |
class add/remove on every link — mozilla blog (95 kB) |
10.3 µs |
137 µs (13.5x ±2%) |
class add/remove on every link — whatwg spec (235 kB) |
10.1 µs |
178 µs (17.7x ±2%) |
drop tags with content (remove) — daring fireball (10 kB) |
23.2 µs |
115 µs (5.0x ±6%) |
drop tags with content (remove) — ars technica (56 kB) |
112 µs |
453 µs (4.1x ±1%) |
drop tags with content (remove) — mozilla blog (95 kB) |
252 µs |
886 µs (3.6x ±1%) |
drop tags with content (remove) — whatwg spec (235 kB) |
650 µs |
2.2 ms (3.4x ±2%) |
unwrap tags keep content (strip_tags) — daring fireball (10 kB) |
24.4 µs |
132 µs (5.5x ±2%) |
unwrap tags keep content (strip_tags) — ars technica (56 kB) |
120 µs |
525 µs (4.4x ±2%) |
unwrap tags keep content (strip_tags) — mozilla blog (95 kB) |
261 µs |
973 µs (3.8x ±4%) |
unwrap tags keep content (strip_tags) — whatwg spec (235 kB) |
669 µs |
2.59 ms (3.9x ±3%) |
replace body inner HTML — daring fireball (10 kB) |
2.12 µs |
17.5 µs (8.3x ±2%) |
replace body inner HTML — ars technica (56 kB) |
7.8 µs |
57.4 µs (7.4x ±6%) |
replace body inner HTML — mozilla blog (95 kB) |
11.7 µs |
123 µs (10.6x ±2%) |
replace body inner HTML — whatwg spec (235 kB) |
35.3 µs |
363 µs (10.3x ±2%) |
replace body text — daring fireball (10 kB) |
1.13 µs |
14.7 µs (13.0x ±2%) |
replace body text — ars technica (56 kB) |
6.4 µs |
54.1 µs (8.5x ±1%) |
replace body text — mozilla blog (95 kB) |
10.5 µs |
118 µs (11.3x ±1%) |
replace body text — whatwg spec (235 kB) |
33 µs |
357 µs (10.9x ±1%) |
walk every descendant — daring fireball (10 kB) |
3.37 µs |
75.7 µs (22.5x ±1%) |
walk every descendant — ars technica (56 kB) |
13.3 µs |
278 µs (20.9x ±3%) |
walk every descendant — mozilla blog (95 kB) |
28.5 µs |
616 µs (21.6x ±3%) |
walk every descendant — whatwg spec (235 kB) |
97.5 µs |
2.13 ms (21.9x ±1%) |
fluent jQuery-style chain — daring fireball (10 kB) |
1.74 µs |
82.6 µs (47.6x ±1%) |
fluent jQuery-style chain — ars technica (56 kB) |
5.34 µs |
175 µs (32.9x ±1%) |
fluent jQuery-style chain — mozilla blog (95 kB) |
8.09 µs |
238 µs (29.5x ±3%) |
fluent jQuery-style chain — whatwg spec (235 kB) |
17.3 µs |
302 µs (17.5x ±3%) |
extract every link — daring fireball (10 kB) |
13.5 µs |
314 µs (23.4x ±21%) |
extract every link — ars technica (56 kB) |
45.7 µs |
759 µs (16.7x ±18%) |
extract every link — mozilla blog (95 kB) |
75.5 µs |
994 µs (13.2x ±14%) |
extract every link — whatwg spec (235 kB) |
66.4 µs |
950 µs (14.4x ±7%) |
absolutize every link — daring fireball (10 kB) |
39.4 µs |
410 µs (10.5x ±6%) |
absolutize every link — ars technica (56 kB) |
91.8 µs |
989 µs (10.8x ±3%) |
absolutize every link — mozilla blog (95 kB) |
157 µs |
1.44 ms (9.3x ±3%) |
absolutize every link — whatwg spec (235 kB) |
224 µs |
1.84 ms (8.3x ±5%) |
rewrite every link — daring fireball (10 kB) |
3 µs |
378 µs (127x ±25%) |
rewrite every link — ars technica (56 kB) |
10.7 µs |
751 µs (70.1x ±6%) |
rewrite every link — mozilla blog (95 kB) |
19.9 µs |
1.04 ms (52.2x ±5%) |
rewrite every link — whatwg spec (235 kB) |
33.3 µs |
1.2 ms (36.2x ±4%) |
social-card extraction — head |
1.83 µs |
64.1 µs (35.2x ±3%) |
social-card extraction — article 8 KiB |
21.9 µs |
133 µs (6.1x ±1%) |
extract @href per match — daring fireball (10 kB) |
3.57 µs |
237 µs (66.6x ±5%) |
extract @href per match — ars technica (56 kB) |
9.06 µs |
542 µs (59.9x ±5%) |
extract @href per match — mozilla blog (95 kB) |
12.5 µs |
764 µs (61.3x ±4%) |
extract @href per match — whatwg spec (235 kB) |
14.4 µs |
939 µs (65.2x ±12%) |
extract text per match — daring fireball (10 kB) |
3.44 µs |
143 µs (41.6x ±12%) |
extract text per match — ars technica (56 kB) |
8.73 µs |
339 µs (38.8x ±5%) |
extract text per match — mozilla blog (95 kB) |
14.6 µs |
507 µs (34.7x ±15%) |
extract text per match — whatwg spec (235 kB) |
14.6 µs |
598 µs (40.9x ±14%) |
extract URL hints — base_url / get_base_url |
1.17 µs |
13 µs (11.2x ±3%) |
extract URL hints — meta_refresh / get_meta_refresh |
1.21 µs |
20.9 µs (17.3x ±3%) |
extract filtered page links — daring fireball (10 kB) |
144 µs |
519 µs (3.7x ±16%) |
extract filtered page links — ars technica (56 kB) |
363 µs |
1.36 ms (3.8x ±10%) |
extract filtered page links — mozilla blog (95 kB) |
619 µs |
2.15 ms (3.5x ±8%) |
extract filtered page links — whatwg spec (235 kB) |
1.06 ms |
3.19 ms (3.0x ±13%) |
The whole shared surface – chaining a select/filter/read, setting content, bulk-editing tags, and reading a value off every match – runs several to hundreds of times faster because the wrapper edits its native tree in C and skips a redundant de-duplication when a chain starts from a single node, where pyquery drives lxml under its jQuery-style wrapper.
How to migrate¶
Build a Query from a parsed document and call it with a selector; the method chains port
almost name for name:
from turbohtml import parse
from turbohtml.query import Query
query = Query(parse("<div><a href='/u'>l</a><a>m</a></div>"))
print(query("a").filter("[href]").eq(0).add_class("seen").attr("href"))
print([anchor.text() for anchor in query("a").items()])
/u
['l', 'm']
turbohtml |
|
|---|---|
|
|
|
|
|
the same names |
|
the same names |
|
the same names |
|
|
|
the same names |
iterating |
|
jQuery |
|
jQuery |
pyquery’s .wrap_all(html) wraps a whole matched set in one new container in place; the node API has two methods for
the shapes that fit a tree model cleanly. wrap_children() boxes every child of a container, and
wrap_siblings() wraps a node and the contiguous run of siblings after it (through an until
node, or to the last sibling), so query("p").wrap_all("<div/>") over a run of adjacent paragraphs becomes
first.wrap_siblings(Element("div"), until=last):
turbohtml |
|
|---|---|
|
|
|
pyquery’s content setters – .html(markup) reparses a matched element’s children and .text(s) replaces them with
one verbatim text node – map onto three element methods. set_inner_html() parses the markup as
a fragment in the element’s context and replaces its children; set_text() replaces them with
one verbatim text node; and insert_adjacent_html() splices a parsed fragment at a DOM position
(the .append(markup) / insertAdjacentHTML shape):
turbohtml |
|
|---|---|
|
|
|
|
|
Gotchas and pitfalls¶
.wrap_allover an arbitrary, non-contiguous set of nodes has no single node-method counterpart (the set has no shared anchor to place the wrapper at); wrap the contiguous run, orappend()the scattered nodes into one new element and place it yourself.pyquery’s network-fetching constructor (
PyQuery(url=...)) is out of scope: fetch with httpx (or any client) and hand the bytes toturbohtml.parse().pyquery exposes lxml’s
.xpath(...)on the fluent wrapper itself; turbohtml’sQueryis CSS-only, so an XPath chain drops to the node-levelxpath()(XPath 1.0) or thefind()grammar viaQuery.items.turbohtml parses to the WHATWG spec, so a malformed document is fixed up exactly as a browser would (implied
<tbody>, reparented<head>content); pyquery’s tree follows lxml’s HTML parser, which can differ on the same broken input.