From parsel

parsel is Scrapy’s extraction-oriented selector library. A Selector wraps a document and every query returns a SelectorList; you pull strings out of it with get() / getall(), reaching text and attribute values through the non-standard ::text and ::attr(name) pseudo-elements. It layers cssselect (CSS-to-XPath translation), lxml/libxml2 (parsing and evaluation), w3lib, and jmespath into one façade, so a single object exposes CSS, XPath, JMESPath over embedded JSON, and regex extraction. It is the workhorse behind Scrapy spiders and is widely used standalone for scraping and data extraction.

turbohtml covers the same extraction ground with a native, spec-compliant HTML5 parser: select(), xpath(), re(), and attribute access return typed Node objects, and the string-shaping helpers (.text, .html, attr()) are ordinary members rather than pseudo-element syntax.

turbohtml vs parsel

Dimension	turbohtml	parsel
Scope	Spec-compliant HTML5 parser plus native CSS/XPath selection, regex extraction, mutation, and serialization	Extraction-only façade over lxml, cssselect, w3lib, and jmespath; no tree-construction of its own
Feature breadth	CSS Selectors Level 4, XPath, regex, JMESPath (via stdlib), markdown/text/minify/sanitize output, DOM mutation	CSS (translated to XPath), XPath, regex, JMESPath over JSON, XML/RSS parsing mode with namespaces
Performance	Compiles a selector once and matches by interned integer atoms; see the table below	Re-translates each .css() to XPath and re-evaluates on libxml2 per call
Typing	Fully typed, ships .pyi stubs; queries return typed Node	Returns SelectorList of strings; typed but string-centric
Dependencies	Self-contained C extension, no runtime dependencies	Requires lxml, cssselect, w3lib, and jmespath
Maintenance	Actively developed	Actively maintained under the Scrapy project

Feature overlap

These map 1:1 and port with a rename:

• CSS selection: css() becomes select() / select_one().

• XPath: xpath() becomes xpath(), xpath_one(), or xpath_iter(), including the namespaces= keyword.

• Regex extraction: re() / re_first() become re() / re_first(), and the attr keyword runs the pattern over an attribute value instead of the text.

• Attribute access: attrib becomes attrs, and ::attr(name) becomes attr().

• Text extraction: ::text becomes text.

• The underlying element: sel.root (an lxml element) becomes the Node itself.

What turbohtml adds

• A WHATWG-conformant HTML5 tree builder. parsel parses through libxml2’s HTML parser, which does not follow the spec’s tree-construction algorithm; turbohtml matches browser parsing on malformed markup.

• CSS Selectors Level 4 evaluated by a native engine, not translated to XPath and handed to libxml2.

• Tree mutation on the same object: prune(), remove(), strip_tags(), set_text(), insert_adjacent_html().

• Output conversions built in: serialize(), to_markdown(), to_text().

• matches() and closest() for testing and walking without a fresh query.

• No third-party runtime dependencies to install or pin.

What parsel has that turbohtml does not

• JMESPath / JSON selectors (jmespath()): no equivalent. Parse the JSON payload with json and query it with the jmespath package yourself.

• An XML parsing mode (Selector(type="xml")) for RSS/Atom and other XML feeds, with namespace registration: no equivalent, since turbohtml is an HTML parser. turbohtml’s xpath() accepts a namespaces= keyword for HTML documents, but it does not parse standalone XML.

• The ::text / ::attr() pseudo-elements themselves: not CSS-standard and not parsed by turbohtml. Read text and attr() off the selected node instead.

Performance

turbohtml compiles a selector against the tree once and matches by comparing interned integer atoms, where parsel translates every .css() to XPath with cssselect and re-evaluates it on libxml2 per call, so a reused query – and pulling the values out of every match, parsel’s whole point – runs tens to hundreds of times faster:

operation	turbohtml	parsel
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	6.44 µs (5.4x)
parse to a tree — wpt small (4 kB)	9.58 µs	30.5 µs (3.2x)
parse to a tree — wpt medium (9.6 kB)	24.1 µs	77.5 µs (3.3x)
parse to a tree — wpt large (92 kB)	209 µs	637 µs (3.1x)
parse to a tree — wpt CJK (124 kB)	408 µs	1.44 ms (3.6x)
parse to a tree — whatwg spec (235 kB)	400 µs	1.26 ms (3.2x)
find every anchor — daring fireball (10 kB)	371 ns	20.7 µs (55.9x)
find every anchor — ars technica (56 kB)	817 ns	47.3 µs (57.9x)
find every anchor — mozilla blog (95 kB)	1.16 µs	67.6 µs (58.2x)
find every anchor — whatwg spec (235 kB)	1.36 µs	104 µs (76.7x)
select div a[href] — daring fireball (10 kB)	610 ns	32.9 µs (54.0x)
select div a[href] — ars technica (56 kB)	1.47 µs	143 µs (97.9x)
select div a[href] — mozilla blog (95 kB)	2.1 µs	840 µs (400x)
select div a[href] — whatwg spec (235 kB)	1.78 µs	1.37 ms (771x)
select div:has(a) — daring fireball (10 kB)	254 ns	10.9 µs (43.1x)
select div:has(a) — ars technica (56 kB)	1.24 µs	40.2 µs (32.3x)
select div:has(a) — mozilla blog (95 kB)	8.99 µs	84.4 µs (9.4x)
select div:has(a) — whatwg spec (235 kB)	5.8 µs	145 µs (25.0x)
find by text content — daring fireball (10 kB)	24.4 µs	45.9 µs (1.9x)
find by text content — ars technica (56 kB)	178 µs	237 µs (1.4x)
find by text content — mozilla blog (95 kB)	287 µs	486 µs (1.7x)
find by text content — whatwg spec (235 kB)	560 µs	1.23 ms (2.2x)
collect visible text — daring fireball (10 kB)	2.7 µs	294 µs (109x)
collect visible text — ars technica (56 kB)	13 µs	1.32 ms (102x)
collect visible text — mozilla blog (95 kB)	22.4 µs	3.18 ms (142x)
collect visible text — whatwg spec (235 kB)	75.4 µs	11.9 ms (158x)
serialize a parsed tree — daring fireball (10 kB)	7.02 µs	33.7 µs (4.8x)
serialize a parsed tree — ars technica (56 kB)	38.8 µs	167 µs (4.3x)
serialize a parsed tree — mozilla blog (95 kB)	76.3 µs	364 µs (4.8x)
serialize a parsed tree — whatwg spec (235 kB)	195 µs	680 µs (3.5x)
extract every link — daring fireball (10 kB)	8.49 µs	66.1 µs (7.8x)
extract every link — ars technica (56 kB)	27.6 µs	144 µs (5.3x)
extract every link — mozilla blog (95 kB)	50.1 µs	216 µs (4.3x)
extract every link — whatwg spec (235 kB)	84.6 µs	270 µs (3.2x)
extract @href per match — daring fireball (10 kB)	2.49 µs	65.1 µs (26.2x)
extract @href per match — ars technica (56 kB)	6.29 µs	142 µs (22.6x)
extract @href per match — mozilla blog (95 kB)	8.51 µs	212 µs (24.9x)
extract @href per match — whatwg spec (235 kB)	9.05 µs	275 µs (30.4x)
extract text per match — daring fireball (10 kB)	2.75 µs	66.6 µs (24.2x)
extract text per match — ars technica (56 kB)	6.39 µs	129 µs (20.2x)
extract text per match — mozilla blog (95 kB)	10.3 µs	189 µs (18.5x)
extract text per match — whatwg spec (235 kB)	10.8 µs	247 µs (22.9x)
extract URL hints — base_url / get_base_url	1.27 µs	12.9 µs (10.2x)
extract URL hints — meta_refresh / get_meta_refresh	1.29 µs	12.6 µs (9.9x)
xpath_path for every element — daring fireball (10 kB)	17.7 µs	103 µs (5.9x)
xpath_path for every element — ars technica (56 kB)	93.1 µs	539 µs (5.8x)
xpath_path for every element — mozilla blog (95 kB)	264 µs	1.37 ms (5.2x)
xpath_path for every element — whatwg spec (235 kB)	2.08 ms	9.66 ms (4.7x)
CSS selector to XPath 1.0 — type	239 ns	1.05 µs (4.5x)
CSS selector to XPath 1.0 — compound	372 ns	12.8 µs (34.4x)
CSS selector to XPath 1.0 — structural	296 ns	12.1 µs (40.9x)
CSS selector to XPath 1.0 — complex	567 ns	22.3 µs (39.4x)
CSS selector to XPath 1.0 — group	452 ns	14.5 µs (32.2x)
XPath feature surface (9.6 kB) — //div	1.77 µs	37.1 µs (21.0x)
XPath feature surface (9.6 kB) — //a[@href]	368 ns	5.66 µs (15.4x)
XPath feature surface (9.6 kB) — //div//a[@href]	1.47 µs	11.8 µs (8.1x)
XPath feature surface (9.6 kB) — /html/body/div	806 ns	18.6 µs (23.1x)
XPath feature surface (9.6 kB) — //div//a[1]	10.2 µs	11.9 µs (1.2x)
XPath feature surface (9.6 kB) — //a[contains(@href, ‘/’)]	363 ns	5.97 µs (16.5x)
XPath feature surface (9.6 kB) — //div[position() <= 3]	5.47 µs	30 µs (5.5x)
XPath feature surface (9.6 kB) — //a/ancestor::div	371 ns	4.45 µs (12.1x)
XPath feature surface (9.6 kB) — //a | //span	606 ns	5.37 µs (8.9x)
XPath feature surface (9.6 kB) — //*[local-name() = ‘a’]	4.35 µs	16.4 µs (3.8x)
XPath feature surface (9.6 kB) — count(//a)	392 ns	5.17 µs (13.2x)
XPath feature surface (9.6 kB) — //a[@href=$x] (variable)	528 ns	5.91 µs (11.2x)
XPath feature surface (9.6 kB) — //a[re:test(@href, …)] (EXSLT)	360 ns	5.81 µs (16.2x)
XPath feature surface (9.6 kB) — set:distinct(//a) (EXSLT)	415 ns	4.23 µs (10.2x)
XPath feature surface (9.6 kB) — //a/@href (smart_strings)	466 ns	4.07 µs (8.8x)
XPath feature surface (9.6 kB) — ext(//a) (extensions)	878 ns	5.56 µs (6.4x)
XPath feature surface (9.6 kB) — ext(//a)/@href (node-set extension)	904 ns	4.7 µs (5.2x)
XPath feature surface (9.6 kB) — //svg:rect (namespaces=)	555 ns	4.45 µs (8.1x)
XPath feature surface (9.6 kB) — $rows/div (node-set variable)	2.54 µs	18.2 µs (7.2x)
XPath feature surface (9.6 kB) — //a[@href] (precompiled, reused)	333 ns	2.8 µs (8.4x)
extract filtered page links — daring fireball (10 kB)	127 µs	67.7 µs (0.6x)
extract filtered page links — ars technica (56 kB)	310 µs	153 µs (0.5x)
extract filtered page links — mozilla blog (95 kB)	514 µs	216 µs (0.5x)
extract filtered page links — whatwg spec (235 kB)	889 µs	276 µs (0.4x)

How to migrate

Replace Selector(text=html) with turbohtml.parse(), then swap the string-extraction calls for typed-node access. The regex helpers re() and re_first() carry over directly, including their attr keyword.

parsel	turbohtml
Selector(text=html)	turbohtml.parse()
parsel.selector.Selector.css(), parsel.selector.Selector.xpath()	select(), xpath()
sel.css("a").get() (outer HTML)	select_one() then html
sel.css("a::text").get(), .getall()	text off each selected node
sel.css("a::attr(href)").get(), .getall()	attr() off each selected node
sel.xpath("//a/@href").getall()	xpath() (already yields the values)
parsel.selector.Selector.attrib	attrs
parsel.selector.Selector.re(), parsel.selector.Selector.re_first()	re(), re_first()
sel.css("a::attr(href)").re(pattern)	re() with attr="href"
sel.root (an lxml element)	the Node itself

doc = parse('<a href="/x">home</a><a href="/y">about</a>')
print([a.attr("href") for a in doc.select("a")])
print(doc.select_one("a").text)
print(doc.xpath("//a/@href"))
print([a.re_first(r"\w+") for a in doc.select("a")])
print(doc.select_one("a").re_first(r"/(\w+)", attr="href"))

['/x', '/y']
home
['/x', '/y']
['home', 'about']
x

Gotchas and pitfalls

• parsel’s ::text and ::attr() pseudo-elements are not CSS standard and turbohtml does not parse them; read text and attr() off the selected node instead.

• get() / getall() return strings; turbohtml returns nodes, so choose .text, .html, attr(), or re() explicitly per call. A turbohtml xpath("//a/@href") already yields the attribute values as strings, so there is no .getall() to chain.

• re() and re_first() run over one node at a time rather than a whole SelectorList; map them across select() to cover every match.

• parsel’s JSON/JMESPath selectors (jmespath()) are not ported; run json/jmespath over parsed JSON yourself.

• parsel returns None (via .get(default=None)) or an empty SelectorList for a miss; turbohtml’s select_one() returns None and select() returns an empty list, so guard the None before reading .text or attr().

• parsel drives libxml2’s non-spec HTML parser, so tree shape on malformed markup can differ from turbohtml’s WHATWG-conformant construction; re-check selectors that relied on libxml2’s tolerant fixups.