From parsel

parsel is Scrapy’s extraction-oriented selector library. A Selector wraps a document and every query returns a SelectorList; you pull strings out of it with get() / getall(), reaching text and attribute values through the non-standard ::text and ::attr(name) pseudo-elements. It layers cssselect (CSS-to-XPath translation), lxml/libxml2 (parsing and evaluation), w3lib, and jmespath into one façade, so a single object exposes CSS, XPath, JMESPath over embedded JSON, and regex extraction. It is the workhorse behind Scrapy spiders and is widely used standalone for scraping and data extraction.

turbohtml covers the same extraction ground with a native, spec-compliant HTML5 parser: select(), xpath(), re(), and attribute access return typed Node objects, and the string-shaping helpers (.text, .html, attr()) are ordinary members rather than pseudo-element syntax.

turbohtml vs parsel

Dimension	turbohtml	parsel
Scope	Spec-compliant HTML5 parser plus native CSS/XPath selection, regex extraction, mutation, and serialization	Extraction-only façade over lxml, cssselect, w3lib, and jmespath; no tree-construction of its own
Feature breadth	CSS Selectors Level 4, XPath, regex, JMESPath (via stdlib), markdown/text/minify/sanitize output, DOM mutation	CSS (translated to XPath), XPath, regex, JMESPath over JSON, XML/RSS parsing mode with namespaces
Performance	Compiles a selector once and matches by interned integer atoms; see the table below	Re-translates each .css() to XPath and re-evaluates on libxml2 per call
Typing	Fully typed, ships .pyi stubs; queries return typed Node	Returns SelectorList of strings; typed but string-centric
Dependencies	Self-contained C extension, no runtime dependencies	Requires lxml, cssselect, w3lib, and jmespath
Maintenance	Actively developed	Actively maintained under the Scrapy project

Feature overlap

These map 1:1 and port with a rename:

• CSS selection: css() becomes select() / select_one().

• XPath: xpath() becomes xpath(), xpath_one(), or xpath_iter(), including the namespaces= keyword.

• Regex extraction: re() / re_first() become re() / re_first(), and the attr keyword runs the pattern over an attribute value instead of the text.

• Attribute access: attrib becomes attrs, and ::attr(name) becomes attr().

• Text extraction: ::text becomes text.

• The underlying element: sel.root (an lxml element) becomes the Node itself.

What turbohtml adds

• A WHATWG-conformant HTML5 tree builder. parsel parses through libxml2’s HTML parser, which does not follow the spec’s tree-construction algorithm; turbohtml matches browser parsing on malformed markup.

• CSS Selectors Level 4 evaluated by a native engine, not translated to XPath and handed to libxml2.

• Tree mutation on the same object: prune(), remove(), strip_tags(), set_text(), insert_adjacent_html().

• Output conversions built in: serialize(), to_markdown(), to_text().

• matches() and closest() for testing and walking without a fresh query.

• No third-party runtime dependencies to install or pin.

What parsel has that turbohtml does not

• JMESPath / JSON selectors (jmespath()): no equivalent. Parse the JSON payload with json and query it with the jmespath package yourself.

• An XML parsing mode (Selector(type="xml")) for RSS/Atom and other XML feeds, with namespace registration: no equivalent, since turbohtml is an HTML parser. turbohtml’s xpath() accepts a namespaces= keyword for HTML documents, but it does not parse standalone XML.

• The ::text / ::attr() pseudo-elements themselves: not CSS-standard and not parsed by turbohtml. Read text and attr() off the selected node instead.

Performance

turbohtml compiles a selector against the tree once and matches by comparing interned integer atoms, where parsel translates every .css() to XPath with cssselect and re-evaluates it on libxml2 per call, so a reused query – and pulling the values out of every match, parsel’s whole point – runs tens to hundreds of times faster:

operation	turbohtml	parsel
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	6.54 µs (5.5x)
parse to a tree — wpt small (4 kB)	9.57 µs	30.7 µs (3.3x)
parse to a tree — wpt medium (9.6 kB)	24.2 µs	77 µs (3.2x)
parse to a tree — wpt large (92 kB)	207 µs	632 µs (3.1x)
parse to a tree — wpt CJK (124 kB)	407 µs	1.49 ms (3.7x)
parse to a tree — whatwg spec (235 kB)	405 µs	1.31 ms (3.3x)
find every anchor — daring fireball (10 kB)	375 ns	21.1 µs (56.3x)
find every anchor — ars technica (56 kB)	828 ns	47.9 µs (57.9x)
find every anchor — mozilla blog (95 kB)	1.18 µs	71.6 µs (60.9x)
find every anchor — whatwg spec (235 kB)	1.32 µs	104 µs (79.4x)
select div a[href] — daring fireball (10 kB)	639 ns	32.9 µs (51.5x)
select div a[href] — ars technica (56 kB)	1.46 µs	146 µs (99.7x)
select div a[href] — mozilla blog (95 kB)	2.07 µs	828 µs (401x)
select div a[href] — whatwg spec (235 kB)	1.74 µs	1.41 ms (812x)
select div:has(a) — daring fireball (10 kB)	256 ns	11 µs (42.8x)
select div:has(a) — ars technica (56 kB)	1.28 µs	40.9 µs (32.0x)
select div:has(a) — mozilla blog (95 kB)	8.97 µs	85 µs (9.5x)
select div:has(a) — whatwg spec (235 kB)	6.08 µs	142 µs (23.4x)
find by text content — daring fireball (10 kB)	42.9 µs	47.8 µs (1.2x)
find by text content — ars technica (56 kB)	431 µs	353 µs (0.9x)
find by text content — mozilla blog (95 kB)	329 µs	736 µs (2.3x)
find by text content — whatwg spec (235 kB)	638 µs	2 ms (3.2x)
collect visible text — daring fireball (10 kB)	2.69 µs	293 µs (109x)
collect visible text — ars technica (56 kB)	13.4 µs	1.34 ms (101x)
collect visible text — mozilla blog (95 kB)	23.3 µs	3.25 ms (140x)
collect visible text — whatwg spec (235 kB)	76.8 µs	12.3 ms (161x)
serialize a parsed tree — daring fireball (10 kB)	7.14 µs	34 µs (4.8x)
serialize a parsed tree — ars technica (56 kB)	40.7 µs	167 µs (4.2x)
serialize a parsed tree — mozilla blog (95 kB)	79.2 µs	368 µs (4.7x)
serialize a parsed tree — whatwg spec (235 kB)	209 µs	708 µs (3.4x)
extract every link — daring fireball (10 kB)	8.66 µs	66.8 µs (7.8x)
extract every link — ars technica (56 kB)	27.7 µs	148 µs (5.4x)
extract every link — mozilla blog (95 kB)	51.5 µs	214 µs (4.2x)
extract every link — whatwg spec (235 kB)	84.8 µs	273 µs (3.3x)
extract @href per match — daring fireball (10 kB)	2.58 µs	67.9 µs (26.3x)
extract @href per match — ars technica (56 kB)	6.65 µs	148 µs (22.3x)
extract @href per match — mozilla blog (95 kB)	9.23 µs	313 µs (34.0x)
extract @href per match — whatwg spec (235 kB)	9.81 µs	356 µs (36.3x)
extract text per match — daring fireball (10 kB)	2.69 µs	117 µs (43.7x)
extract text per match — ars technica (56 kB)	6.68 µs	292 µs (43.7x)
extract text per match — mozilla blog (95 kB)	10.2 µs	471 µs (46.1x)
extract text per match — whatwg spec (235 kB)	10.9 µs	260 µs (24.0x)
extract URL hints — base_url / get_base_url	1.25 µs	12.4 µs (10.0x)
extract URL hints — meta_refresh / get_meta_refresh	1.31 µs	12.8 µs (9.8x)
xpath_path for every element — daring fireball (10 kB)	17.8 µs	105 µs (5.9x)
xpath_path for every element — ars technica (56 kB)	92.5 µs	544 µs (5.9x)
xpath_path for every element — mozilla blog (95 kB)	262 µs	1.49 ms (5.7x)
xpath_path for every element — whatwg spec (235 kB)	2.09 ms	6 ms (2.9x)
CSS selector to XPath 1.0 — type	264 ns	984 ns (3.8x)
CSS selector to XPath 1.0 — compound	361 ns	12.7 µs (35.2x)
CSS selector to XPath 1.0 — structural	325 ns	11.9 µs (36.6x)
CSS selector to XPath 1.0 — complex	534 ns	22.4 µs (42.0x)
CSS selector to XPath 1.0 — group	565 ns	15.2 µs (26.9x)
XPath feature surface (9.6 kB) — //div	1.75 µs	43.5 µs (25.0x)
XPath feature surface (9.6 kB) — //a[@href]	369 ns	6.78 µs (18.4x)
XPath feature surface (9.6 kB) — //div//a[@href]	1.49 µs	14.1 µs (9.5x)
XPath feature surface (9.6 kB) — /html/body/div	784 ns	22.4 µs (28.6x)
XPath feature surface (9.6 kB) — //div//a[1]	9.51 µs	11.7 µs (1.3x)
XPath feature surface (9.6 kB) — //a[contains(@href, ‘/’)]	383 ns	5.76 µs (15.1x)
XPath feature surface (9.6 kB) — //div[position() <= 3]	5.41 µs	28.1 µs (5.2x)
XPath feature surface (9.6 kB) — //a/ancestor::div	379 ns	4.05 µs (10.7x)
XPath feature surface (9.6 kB) — //a | //span	639 ns	4.7 µs (7.4x)
XPath feature surface (9.6 kB) — //*[local-name() = ‘a’]	4.61 µs	16.2 µs (3.6x)
XPath feature surface (9.6 kB) — count(//a)	387 ns	4.92 µs (12.8x)
XPath feature surface (9.6 kB) — //a[@href=$x] (variable)	500 ns	5.92 µs (11.9x)
XPath feature surface (9.6 kB) — //a[re:test(@href, …)] (EXSLT)	367 ns	5.75 µs (15.7x)
XPath feature surface (9.6 kB) — set:distinct(//a) (EXSLT)	420 ns	4.21 µs (10.1x)
XPath feature surface (9.6 kB) — //a/@href (smart_strings)	473 ns	4.05 µs (8.6x)
XPath feature surface (9.6 kB) — ext(//a) (extensions)	904 ns	5.63 µs (6.3x)
XPath feature surface (9.6 kB) — ext(//a)/@href (node-set extension)	952 ns	4.76 µs (5.0x)
XPath feature surface (9.6 kB) — //svg:rect (namespaces=)	567 ns	4.35 µs (7.7x)
XPath feature surface (9.6 kB) — $rows/div (node-set variable)	2.48 µs	18.2 µs (7.4x)
XPath feature surface (9.6 kB) — //a[@href] (precompiled, reused)	316 ns	2.79 µs (8.9x)
extract filtered page links — daring fireball (10 kB)	117 µs	65.2 µs (0.6x)
extract filtered page links — ars technica (56 kB)	280 µs	146 µs (0.6x)
extract filtered page links — mozilla blog (95 kB)	471 µs	213 µs (0.5x)
extract filtered page links — whatwg spec (235 kB)	786 µs	268 µs (0.4x)

How to migrate

Replace Selector(text=html) with turbohtml.parse(), then swap the string-extraction calls for typed-node access. The regex helpers re() and re_first() carry over directly, including their attr keyword.

parsel	turbohtml
Selector(text=html)	turbohtml.parse()
parsel.selector.Selector.css(), parsel.selector.Selector.xpath()	select(), xpath()
sel.css("a").get() (outer HTML)	select_one() then html
sel.css("a::text").get(), .getall()	text off each selected node
sel.css("a::attr(href)").get(), .getall()	attr() off each selected node
sel.xpath("//a/@href").getall()	xpath() (already yields the values)
parsel.selector.Selector.attrib	attrs
parsel.selector.Selector.re(), parsel.selector.Selector.re_first()	re(), re_first()
sel.css("a::attr(href)").re(pattern)	re() with attr="href"
sel.root (an lxml element)	the Node itself

doc = parse('<a href="/x">home</a><a href="/y">about</a>')
print([a.attr("href") for a in doc.select("a")])
print(doc.select_one("a").text)
print(doc.xpath("//a/@href"))
print([a.re_first(r"\w+") for a in doc.select("a")])
print(doc.select_one("a").re_first(r"/(\w+)", attr="href"))

['/x', '/y']
home
['/x', '/y']
['home', 'about']
x

Gotchas and pitfalls

• parsel’s ::text and ::attr() pseudo-elements are not CSS standard and turbohtml does not parse them; read text and attr() off the selected node instead.

• get() / getall() return strings; turbohtml returns nodes, so choose .text, .html, attr(), or re() explicitly per call. A turbohtml xpath("//a/@href") already yields the attribute values as strings, so there is no .getall() to chain.

• re() and re_first() run over one node at a time rather than a whole SelectorList; map them across select() to cover every match.

• parsel’s JSON/JMESPath selectors (jmespath()) are not ported; run json/jmespath over parsed JSON yourself.

• parsel returns None (via .get(default=None)) or an empty SelectorList for a miss; turbohtml’s select_one() returns None and select() returns an empty list, so guard the None before reading .text or attr().

• parsel drives libxml2’s non-spec HTML parser, so tree shape on malformed markup can differ from turbohtml’s WHATWG-conformant construction; re-check selectors that relied on libxml2’s tolerant fixups.