From w3lib¶

w3lib is the Scrapy project’s grab-bag of stateless web utilities: character-reference resolution, regex tag/comment stripping, URL canonicalization and query cleaning, and response-encoding detection. It carries no parser and no tree; every helper is a pure-Python function over a string, bytes, or urllib split result, which is why it ships as Scrapy’s low-level layer and shows up in scrapers that need one canonical URL form or a quick entity decode without pulling in a full DOM.

turbohtml covers the same ground from the other direction. The w3lib.html text and entity helpers map onto the WHATWG tree turbohtml already builds, so a regex tag strip becomes a real parse plus a text read; the w3lib.url canonicalization surface maps onto the turbohtml.extract URL helpers, whose canonical form is the WHATWG URL standard serialization instead of w3lib’s urllib re-encoding. Only w3lib’s HTTP-header and file/data-URI plumbing stays outside turbohtml’s scope.

turbohtml vs w3lib¶

Dimension	turbohtml	w3lib
Scope	WHATWG HTML parser, DOM tree, selectors, serializer, URL/extract and encoding detection	Stateless string, URL, and encoding utilities for scraping, no parser or tree
Feature breadth	Broader on HTML: real tree, `strip_tags`, `remove`, document URL hints, sanitizing `clean`	Broader on transport: HTTP headers, file/data URIs, in-place query-parameter mutation
Performance	C extension; 2x-6x on the shared URL batch, several times faster on entity-heavy text	Pure-Python regex and `urllib` re-encoding
Typing	Fully annotated with bundled `.pyi` stubs for the C extension	Typed, `py.typed`
Dependencies	Compiled C extension, no Python runtime dependencies	Pure Python, standard library only
Maintenance	Actively developed	Mature and stable under the Scrapy org

Feature overlap¶

These map 1:1 and port directly:

w3lib.html.replace_entities() -> turbohtml.unescape() (same character-reference resolution).
w3lib.html.remove_tags() -> turbohtml.parse() then text, or strip_tags() to unwrap only a chosen set while keeping the rest of the document.
w3lib.html.remove_comments() -> text (comments never appear in text).
w3lib.html.remove_tags_with_content() -> remove() (drops the tag and its subtree).
w3lib.html.get_base_url() -> base_url().
w3lib.html.get_meta_refresh() -> meta_refresh().
w3lib.url.canonicalize_url() -> turbohtml.extract.normalize_url() with UrlCleaning.w3lib().
w3lib.url.url_query_cleaner() -> UrlCleaning(query_allow=...) / UrlCleaning(query_deny=...).
w3lib.url.safe_url_string() -> turbohtml.extract.clean_url() (also validates) or normalize_url().
w3lib.url.is_url -> turbohtml.extract.clean_url(text) is not None.

What turbohtml adds¶

A real WHATWG tree behind every text operation: nested and malformed markup is parsed the way a browser does, not matched as <...> spans, and text returns decoded characters rather than leaving entities encoded.
Selector-based tree editing (remove(), strip_tags()) that reshapes the document in place, where w3lib only returns strings.
Sanitizing output via turbohtml.clean when the goal is safe HTML rather than reshaping a tree, which w3lib has no concept of.
WHATWG-standard URL serialization from turbohtml.extract, including default-port stripping, .. segment resolution, and automatic removal of known tracking parameters (utm_*, gclid, …) that canonicalize_url never drops.
Encoding detection from bytes through turbohtml.detect.detect().

What w3lib has that turbohtml does not¶

w3lib.url.add_or_replace_parameter / add_or_replace_parameters — building a URL by setting a query value. No equivalent; turbohtml’s UrlCleaning filters parameters but does not add or rewrite them.
w3lib.url.url_query_parameter — reading a single query value out of a URL. No equivalent; use urllib.parse for extraction.
File and data URI helpers (file_uri_to_path, path_to_file_uri, any_to_uri, parse_data_uri). No equivalent, these are outside turbohtml’s scope.
HTTP-header helpers (basic_auth_header, headers_dict_to_raw). No equivalent.
w3lib.encoding’s full decode chain (html_to_unicode, http_content_type_encoding, html_body_declared_encoding, read_bom) that returns decoded text. turbohtml.detect.detect() answers the encoding-label question from bytes but does not chain an HTTP Content-Type header into the decision or hand back the decoded string.

Performance¶

The URL surface is 2x-6x faster over a shared 100-URL batch, and the entity and tag helpers each run a structure-aware C pass that still beats w3lib’s regex on entity-heavy input:

operation	turbohtml	w3lib
unescape — tiny plain (64 B)	34 ns	265 ns (7.8x)
unescape — medium dense refs (4 KiB)	7.09 µs	114 µs (16.1x)
unescape — numeric refs (4 KiB)	4.97 µs	90.9 µs (18.3x)
unescape — book HTML, real refs (4 MiB)	3.34 ms	13.6 ms (4.1x)
unescape — escaped book HTML (5 MiB)	1.76 ms	35.3 ms (20.2x)
unescape — dense refs (4 MiB)	9.67 ms	121 ms (12.6x)
unescape — UCS-2 refs (4 MiB)	2.75 ms	27.8 ms (10.2x)
drop tags with content (remove) — daring fireball (10 kB)	25.3 µs	24.3 µs (1.0x)
drop tags with content (remove) — ars technica (56 kB)	125 µs	101 µs (0.9x)
drop tags with content (remove) — mozilla blog (95 kB)	283 µs	159 µs (0.6x)
drop tags with content (remove) — whatwg spec (235 kB)	687 µs	327 µs (0.5x)
unwrap tags keep content (strip_tags) — daring fireball (10 kB)	25.8 µs	70.7 µs (2.8x)
unwrap tags keep content (strip_tags) — ars technica (56 kB)	127 µs	322 µs (2.6x)
unwrap tags keep content (strip_tags) — mozilla blog (95 kB)	286 µs	629 µs (2.3x)
unwrap tags keep content (strip_tags) — whatwg spec (235 kB)	742 µs	1.74 ms (2.4x)
extract URL hints — base_url / get_base_url	1.25 µs	7.44 µs (6.0x)
extract URL hints — meta_refresh / get_meta_refresh	1.31 µs	6.54 µs (5.0x)
clean and normalize 100 URLs — clean 100 URLs	262 µs	628 µs (2.4x)
clean and normalize 100 URLs — normalize 100 URLs	216 µs	1.27 ms (5.9x)

How to migrate¶

Swap the imports for the turbohtml entry points:

from turbohtml import parse, unescape
from turbohtml.extract import UrlCleaning, clean_url, normalize_url

Then map each call:

w3lib	turbohtml
`w3lib.html.replace_entities()`	`turbohtml.unescape()`
`w3lib.html.remove_tags()`	`turbohtml.parse()` then `text`
`w3lib.html.remove_comments()`	`text` (comments never appear in text)
`w3lib.html.remove_tags_with_content()`	`remove()`
`w3lib.html.get_base_url()`	`base_url()`
`w3lib.html.get_meta_refresh()`	`meta_refresh()`
`w3lib.url.canonicalize_url()`	`turbohtml.extract.normalize_url()` with `UrlCleaning.w3lib()`
`w3lib.url.safe_url_string()`	`turbohtml.extract.clean_url()` (also validates) or `normalize_url()`
`w3lib.url.url_query_cleaner()`	`UrlCleaning(query_allow=...)` / `UrlCleaning(query_deny=...)`
`w3lib.url.is_url`	`turbohtml.extract.clean_url(text) is not None`

replace_entities resolves character references the same way turbohtml.unescape() does, so it is a drop-in; w3lib.html.replace_entities("café & co") returns the same string this prints:

from turbohtml import unescape

print(unescape("caf&eacute; &amp; co"))

café & co

The tag and comment strippers map onto parsing to a real tree and reading its text. remove_tags becomes turbohtml.parse() followed by text, and remove_comments needs nothing extra because comments never appear in text:

from turbohtml import parse

print(parse("<p>Tom &amp; Jerry <b>says</b> hi</p><!--note-->").text)

Tom & Jerry says hi

remove_tags_with_content, which drops a tag together with its subtree, is remove(): remove_tags_with_content(html, which_ones=("script",)) becomes parse(html).remove("script"), editing the tree in place rather than returning a string. When the goal is to drop only some tags while keeping the rest of the document as HTML (remove_tags with which_ones), unwrap them with strip_tags(), which keeps each match’s content. Reach for turbohtml.clean instead when the goal is producing safe HTML rather than reshaping a tree.

The two helpers that read a document’s own URL hints map to the base_url() and meta_refresh() methods on the parsed document. Each takes the fallback base URL w3lib calls baseurl and resolves the hint against it:

from turbohtml import parse

doc = parse('<base href="/sub/"><meta http-equiv=refresh content="5; url=next.html">')
print(doc.base_url("http://site.com/"))
print(doc.meta_refresh("http://site.com/"))

http://site.com/sub/
(5.0, 'http://site.com/next.html')

canonicalize_url becomes turbohtml.extract.normalize_url() with the UrlCleaning.w3lib() preset, which mirrors w3lib’s fragment dropping; url_query_cleaner’s keep/remove parameter lists become the query_allow and query_deny fields on the same config:

from turbohtml.extract import UrlCleaning, normalize_url

print(normalize_url("http://www.example.com/do?c=3&b=5&b=2&a=50#frag", UrlCleaning.w3lib()))
print(
    normalize_url("http://x.example/product.html?id=200&foo=bar&name=wired", UrlCleaning(query_allow=frozenset({"id"})))
)

http://www.example.com/do?a=50&b=2&b=5&c=3
http://x.example/product.html?id=200

Gotchas and pitfalls¶

remove_tags strips angle brackets with a regular expression and leaves entities encoded (Tom & Jerry), while text runs the WHATWG tree builder and returns decoded characters (Tom & Jerry). turbohtml parses malformed and nested markup the way a browser does rather than matching <...> spans, so the two diverge on inputs a regex misreads.
remove_tags_with_content edits the tree rather than returning a string: remove() drops the matches in place, and text() then reads what is left, so a one-line w3lib call becomes a parse-edit-read sequence.
Over the 253 URLs in w3lib’s own test suite the two canonicalizers return identical output for 88% of inputs; every divergence is the URL standard’s form winning over a urllib legacy form. turbohtml keeps a valueless parameter as ?q where w3lib appends ?q=, percent-encodes a query space as %20 where w3lib emits +, leaves , ( ) raw in queries (outside the WHATWG query percent-encode set) where w3lib escapes them, splits parameters on & only (w3lib also splits the legacy ;), strips a default :80/:443 port and resolves .. segments where w3lib keeps both.
normalize_url() always removes known tracking parameters (utm_*, gclid, …); canonicalize_url keeps them. List one in UrlCleaning(query_allow=...) when it must survive.
add_or_replace_parameter, the response-encoding decode chain (turbohtml.detect.detect() answers the encoding question from bytes but does not return decoded text), and the HTTP-header and file/data-URI helpers have no equivalent here.