From w3lib

w3lib latest releasew3lib supported Pythonsw3lib licensew3lib monthly downloadsw3lib total downloadsw3lib GitHub starsw3lib last commit

w3lib is the Scrapy project’s grab-bag of stateless web utilities: character-reference resolution, regex tag/comment stripping, URL canonicalization and query cleaning, and response-encoding detection. It carries no parser and no tree; every helper is a pure-Python function over a string, bytes, or urllib split result, which is why it ships as Scrapy’s low-level layer and shows up in scrapers that need one canonical URL form or a quick entity decode without pulling in a full DOM.

turbohtml covers the same ground from the other direction. The w3lib.html text and entity helpers map onto the WHATWG tree turbohtml already builds, so a regex tag strip becomes a real parse plus a text read; the w3lib.url canonicalization surface maps onto the turbohtml.extract URL helpers, whose canonical form is the WHATWG URL standard serialization instead of w3lib’s urllib re-encoding. Only w3lib’s HTTP-header and file/data-URI plumbing stays outside turbohtml’s scope.

turbohtml vs w3lib

Dimension

turbohtml

w3lib

Scope

WHATWG HTML parser, DOM tree, selectors, serializer, URL/extract and encoding detection

Stateless string, URL, and encoding utilities for scraping, no parser or tree

Feature breadth

Broader on HTML: real tree, strip_tags, remove, document URL hints, sanitizing clean

Broader on transport: HTTP headers, file/data URIs, in-place query-parameter mutation

Performance

C extension; 4.5x-7.5x on the shared URL batch, several times faster on entity-heavy text

Pure-Python regex and urllib re-encoding

Typing

Fully annotated with bundled .pyi stubs for the C extension

Typed, py.typed

Dependencies

Compiled C extension, no Python runtime dependencies

Pure Python, standard library only

Maintenance

Actively developed

Mature and stable under the Scrapy org

Feature overlap

These map 1:1 and port directly:

What turbohtml adds

  • A real WHATWG tree behind every text operation: nested and malformed markup is parsed the way a browser does, not matched as <...> spans, and text returns decoded characters rather than leaving entities encoded.

  • Selector-based tree editing (remove(), strip_tags()) that reshapes the document in place, where w3lib only returns strings.

  • Sanitizing output via turbohtml.clean when the goal is safe HTML rather than reshaping a tree, which w3lib has no concept of.

  • WHATWG-standard URL serialization from turbohtml.extract, including default-port stripping, .. segment resolution, and automatic removal of known tracking parameters (utm_*, gclid, …) that canonicalize_url never drops.

  • Encoding detection from bytes through turbohtml.detect.detect().

What w3lib has that turbohtml does not

  • w3lib.url.add_or_replace_parameter / add_or_replace_parameters — building a URL by setting a query value. No equivalent; turbohtml’s UrlCleaning filters parameters but does not add or rewrite them.

  • w3lib.url.url_query_parameter — reading a single query value out of a URL. No equivalent; use urllib.parse for extraction.

  • File and data URI helpers (file_uri_to_path, path_to_file_uri, any_to_uri, parse_data_uri). No equivalent, these are outside turbohtml’s scope.

  • HTTP-header helpers (basic_auth_header, headers_dict_to_raw). No equivalent.

  • w3lib.encoding’s full decode chain (html_to_unicode, http_content_type_encoding, html_body_declared_encoding, read_bom) that returns decoded text. turbohtml.detect.detect() answers the encoding-label question from bytes but does not chain an HTTP Content-Type header into the decision or hand back the decoded string.

Performance

The URL surface is 4.5x-7.5x faster over a shared 100-URL batch, and the entity and tag helpers each run a structure-aware C pass that still beats w3lib’s regex on entity-heavy input:

operation

turbohtml

w3lib

unescape — tiny plain (64 B)

35.9 ns

258 ns (7.2x ±5%)

unescape — medium dense refs (4 KiB)

4.92 µs

109 µs (22.2x ±1%)

unescape — numeric refs (4 KiB)

5.17 µs

90.8 µs (17.6x ±1%)

unescape — book HTML, real refs (4 MiB)

1.95 ms

13.1 ms (6.8x ±3%)

unescape — escaped book HTML (5 MiB)

1.7 ms

35.2 ms (20.7x ±2%)

unescape — dense refs (4 MiB)

5.6 ms

118 ms (21.1x ±4%)

unescape — UCS-2 refs (4 MiB)

1.89 ms

27.1 ms (14.4x ±2%)

drop tags with content (remove) — daring fireball (10 kB)1

23.2 µs

23.8 µs (1.1x ±2%)

drop tags with content (remove) — ars technica (56 kB)1

112 µs

96.2 µs (0.9x ±1%)

drop tags with content (remove) — mozilla blog (95 kB)1

252 µs

153 µs (0.7x ±1%)

drop tags with content (remove) — whatwg spec (235 kB)1

650 µs

318 µs (0.5x ±2%)

unwrap tags keep content (strip_tags) — daring fireball (10 kB)

24.4 µs

68.4 µs (2.9x ±2%)

unwrap tags keep content (strip_tags) — ars technica (56 kB)

120 µs

319 µs (2.7x ±4%)

unwrap tags keep content (strip_tags) — mozilla blog (95 kB)

261 µs

630 µs (2.5x ±1%)

unwrap tags keep content (strip_tags) — whatwg spec (235 kB)

669 µs

1.64 ms (2.5x ±3%)

extract URL hints — base_url / get_base_url

1.17 µs

7.36 µs (6.4x ±2%)

extract URL hints — meta_refresh / get_meta_refresh

1.21 µs

6.5 µs (5.4x ±2%)

clean and normalize 100 URLs — clean 100 URLs

200 µs

895 µs (4.5x ±20%)

clean and normalize 100 URLs — normalize 100 URLs

191 µs

1.43 ms (7.5x ±7%)

1 removes the tags with a regular expression over the raw string and never parses, so it cannot honor nesting or the tokenizer rules that decide where an element really ends

How to migrate

Swap the imports for the turbohtml entry points:

from turbohtml import parse, unescape
from turbohtml.extract import UrlCleaning, clean_url, normalize_url

Then map each call:

w3lib

turbohtml

w3lib.html.replace_entities()

turbohtml.unescape()

w3lib.html.remove_tags()

turbohtml.parse() then text

w3lib.html.remove_comments()

text (comments never appear in text)

w3lib.html.remove_tags_with_content()

remove()

w3lib.html.get_base_url()

base_url()

w3lib.html.get_meta_refresh()

meta_refresh()

w3lib.url.canonicalize_url()

turbohtml.extract.normalize_url() with UrlCleaning.w3lib()

w3lib.url.safe_url_string()

turbohtml.extract.clean_url() (also validates) or normalize_url()

w3lib.url.url_query_cleaner()

UrlCleaning(query_allow=...) / UrlCleaning(query_deny=...)

w3lib.url.is_url

turbohtml.extract.clean_url(text) is not None

replace_entities resolves character references the same way turbohtml.unescape() does, so it is a drop-in; w3lib.html.replace_entities("caf&eacute; &amp; co") returns the same string this prints:

from turbohtml import unescape

print(unescape("caf&eacute; &amp; co"))

café & co

The tag and comment strippers map onto parsing to a real tree and reading its text. remove_tags becomes turbohtml.parse() followed by text, and remove_comments needs nothing extra because comments never appear in text:

from turbohtml import parse

print(parse("<p>Tom &amp; Jerry <b>says</b> hi</p><!--note-->").text)

Tom & Jerry says hi

remove_tags_with_content, which drops a tag together with its subtree, is remove(): remove_tags_with_content(html, which_ones=("script",)) becomes parse(html).remove("script"), editing the tree in place rather than returning a string. When the goal is to drop only some tags while keeping the rest of the document as HTML (remove_tags with which_ones), unwrap them with strip_tags(), which keeps each match’s content. Reach for turbohtml.clean instead when the goal is producing safe HTML rather than reshaping a tree.

The two helpers that read a document’s own URL hints map to the base_url() and meta_refresh() methods on the parsed document. Each takes the fallback base URL w3lib calls baseurl and resolves the hint against it:

from turbohtml import parse

doc = parse('<base href="/sub/"><meta http-equiv=refresh content="5; url=next.html">')
print(doc.base_url("http://site.com/"))
print(doc.meta_refresh("http://site.com/"))

http://site.com/sub/
(5.0, 'http://site.com/next.html')

canonicalize_url becomes turbohtml.extract.normalize_url() with the UrlCleaning.w3lib() preset, which mirrors w3lib’s fragment dropping; url_query_cleaner’s keep/remove parameter lists become the query_allow and query_deny fields on the same config:

from turbohtml.extract import UrlCleaning, normalize_url

print(normalize_url("http://www.example.com/do?c=3&b=5&b=2&a=50#frag", UrlCleaning.w3lib()))
print(
    normalize_url("http://x.example/product.html?id=200&foo=bar&name=wired", UrlCleaning(query_allow=frozenset({"id"})))
)

http://www.example.com/do?a=50&b=2&b=5&c=3
http://x.example/product.html?id=200

Gotchas and pitfalls

  • remove_tags strips angle brackets with a regular expression and leaves entities encoded (Tom &amp; Jerry), while text runs the WHATWG tree builder and returns decoded characters (Tom & Jerry). turbohtml parses malformed and nested markup the way a browser does rather than matching <...> spans, so the two diverge on inputs a regex misreads.

  • remove_tags_with_content edits the tree rather than returning a string: remove() drops the matches in place, and text() then reads what is left, so a one-line w3lib call becomes a parse-edit-read sequence.

  • Over the 253 URLs in w3lib’s own test suite the two canonicalizers return identical output for 88% of inputs; every divergence is the URL standard’s form winning over a urllib legacy form. turbohtml keeps a valueless parameter as ?q where w3lib appends ?q=, percent-encodes a query space as %20 where w3lib emits +, leaves , ( ) raw in queries (outside the WHATWG query percent-encode set) where w3lib escapes them, splits parameters on & only (w3lib also splits the legacy ;), strips a default :80/:443 port and resolves .. segments where w3lib keeps both.

  • normalize_url() always removes known tracking parameters (utm_*, gclid, …); canonicalize_url keeps them. List one in UrlCleaning(query_allow=...) when it must survive.

  • add_or_replace_parameter, the response-encoding decode chain (turbohtml.detect.detect() answers the encoding question from bytes but does not return decoded text), and the HTTP-header and file/data-URI helpers have no equivalent here.