From html5lib¶

html5lib is the reference pure-Python implementation of the WHATWG parsing algorithm. It tokenizes a byte or character stream and builds a tree through a treebuilder you select at call time: an xml.etree.ElementTree element by default, or dom, or lxml. On top of the tree it ships treewalkers that convert between representations, a configurable serializer, a filter chain (whitespace collapsing, optional-tag omission, alphabetical attributes, meta-charset injection), and a now-deprecated sanitizer. Because it tracks the spec closely, it is the conformance baseline other parsers are checked against, and it is the html5lib backend BeautifulSoup and others parse through.

turbohtml runs the same WHATWG algorithm in C and covers that ground with one engine. turbohtml.parse() returns a single fully type annotated Document with navigation, find/select/XPath querying, and WHATWG serialization built in, so there is no foreign tree behind a treebuilder choice and no separate walk-and-serialize step.

turbohtml vs html5lib¶

Dimension	turbohtml	html5lib
Scope	WHATWG HTML: parse, tokenize, query, mutate, and serialize in one C engine	WHATWG HTML parse and tokenize into a pluggable third-party tree, plus treewalkers and a serializer
Feature breadth	One typed tree with `find()` grammar, CSS `select()`, XPath, and WHATWG serialization formatters	etree/DOM/lxml treebuilders, treewalkers, serializer filters, deprecated sanitizer; no built-in query API
Performance	C core; parse, tokenize, and fragment parse run 25 to 70 times faster	Pure Python; the reference implementation, tuned for correctness not speed
Typing	Ships `py.typed`; every public method annotated	No inline types and no published stubs
Dependencies	Zero runtime dependencies (self-contained C extension)	Requires `webencodings` and `six`; `lxml` optional for that treebuilder, `chardet` optional for sniffing
Maintenance	Actively developed, single-engine	Mature and stable; the conformance reference, with an infrequent release cadence

Feature overlap¶

These port one-to-one; the calls differ only in name (see the mapping table under How to migrate):

Parsing a full document with the WHATWG algorithm: html5lib.parse maps to turbohtml.parse().
Fragment parsing in a container context: html5lib.parseFragment maps to turbohtml.parse_fragment(), the same WHATWG innerHTML fragment algorithm.
Tokenizing without building a tree: the html5lib tokenizer maps to turbohtml.tokenize() and turbohtml.Tokenizer, yielding Token values tagged by TokenType.
Serializing a tree back to markup: html5lib.serialize maps to serialize() and html.

What turbohtml adds¶

A C engine. Parsing, tokenizing, fragment parsing, and serialization all run in C, 25 to 70 times faster than the pure-Python reference.
A built-in query API. html5lib hands back a tree and stops; you navigate etree/DOM yourself. turbohtml carries find() / find_all(), CSS select(), and xpath() on the tree it returns.
One typed tree. Document, Element, Text, and the other node types are sealed, pattern-matchable, and annotated, instead of a foreign tree chosen through a treebuilder.
Plain tag names with a separate namespace. tag stays plain (div) and the namespace rides on namespace as a Namespace, rather than being folded into a Clark-notation name ({http://www.w3.org/1999/xhtml}div).
WHATWG-conformant serialization by default through Formatter selection, with no serializer object to construct.
Zero runtime dependencies and full typing. No webencodings or six install, and py.typed ships.

What html5lib has that turbohtml does not¶

Pluggable treebuilders. html5lib.parse(s, treebuilder="dom") (or "lxml", or the default "etree") returns whichever tree you name. turbohtml always returns its own typed tree; this is a deliberate clean-break omission. Workaround: if you need an ElementTree or lxml tree specifically, keep html5lib for that call.
Treewalkers. html5lib converts one tree representation into another through html5lib.getTreeWalker. turbohtml has a single representation, so there is nothing to walk between. No equivalent, and none needed.
Serializer filters. html5lib’s serializer chains filters for optional-tag omission, alphabetical attribute order, meta-charset injection, and whitespace. turbohtml serializes WHATWG-conformant output selected by Formatter; it does not expose that filter registry. Workaround: pick the closest Formatter and layout; for optional-tag omission there is no equivalent.
A (deprecated) sanitizer. html5lib ships html5lib.filters.sanitizer, deprecated since 1.1. turbohtml has no sanitizer. Workaround: use a dedicated sanitizer such as nh3 or bleach (see From nh3 and From bleach).
Optional statistical encoding detection. With chardet installed, html5lib’s input stream can guess an encoding from byte frequency when there is no BOM or <meta charset>. turbohtml sniffs only what the WHATWG algorithm reads, then falls back to windows-1252. Workaround: detect with charset-normalizer first and hand turbohtml the decoded str (or bytes with an explicit encoding=).

Performance¶

The algorithm runs in C, so parsing, tokenizing, and fragment parsing run 25 to 70 times faster than the pure-Python implementation (turbohtml.parse_fragment() parses an innerHTML-style snippet in its container context, the same WHATWG fragment algorithm html5lib’s parseFragment() runs):

input	turbohtml	html5lib
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	104 µs (86.7x)
parse to a tree — wpt small (4 kB)	9.57 µs	694 µs (72.6x)
parse to a tree — wpt medium (9.6 kB)	24.2 µs	1.47 ms (60.6x)
parse to a tree — wpt large (92 kB)	207 µs	17 ms (82.0x)
parse to a tree — wpt CJK (124 kB)	407 µs	30.5 ms (74.9x)
parse to a tree — whatwg spec (235 kB)	405 µs	33.1 ms (81.8x)
parse a fragment — table-row fragment (2 kB)	11.4 µs	848 µs (74.3x)
tokenize — typical markup	28.5 µs	817 µs (28.7x)
tokenize — text-heavy prose	535 ns	145 µs (272x)
tokenize — attribute-heavy	18.2 µs	805 µs (44.3x)
tokenize — script-heavy	11.5 µs	497 µs (43.1x)
tokenize — entity-heavy	19.8 µs	1.22 ms (61.8x)
tokenize — wpt tiny (0.6 kB)	1.52 µs	47.6 µs (31.3x)
tokenize — wpt small (4 kB)	11.9 µs	423 µs (35.5x)
tokenize — wpt medium (9.6 kB)	27.3 µs	1.16 ms (42.5x)
tokenize — wpt large (92 kB)	314 µs	9.03 ms (28.8x)
tokenize — wpt CJK (124 kB)	550 µs	22.3 ms (40.6x)
tokenize — whatwg spec (235 kB)	624 µs	19.7 ms (31.7x)
tokenize — ecmascript spec (3 MB)	6.2 ms	180 ms (29.1x)
tokenize — whatwg spec source (7.9 MB)	36.5 ms	850 ms (23.3x)
serialize a parsed tree — daring fireball (10 kB)	7.14 µs	475 µs (66.7x)
serialize a parsed tree — ars technica (56 kB)	40.7 µs	1.95 ms (48.0x)
serialize a parsed tree — mozilla blog (95 kB)	79.2 µs	4.12 ms (52.1x)
serialize a parsed tree — whatwg spec (235 kB)	209 µs	13.8 ms (66.0x)
walk every descendant — daring fireball (10 kB)	3.34 µs	283 µs (84.8x)
walk every descendant — ars technica (56 kB)	13.8 µs	1.15 ms (83.7x)
walk every descendant — mozilla blog (95 kB)	28.2 µs	2.57 ms (91.1x)
walk every descendant — whatwg spec (235 kB)	101 µs	9.88 ms (98.0x)

How to migrate¶

Swap the import. There is no treebuilder to name, since turbohtml always returns its own typed tree:

# html5lib
import html5lib

doc = html5lib.parse("<table><tr><td>x", treebuilder="etree")

from turbohtml import parse

doc = parse("<table><tr><td>x")  # the same tree html5lib and a browser build
print(doc.find("td").text)

API mapping¶

html5lib	turbohtml
`html5lib.parse()`	`turbohtml.parse()`
`html5lib.parse(s, treebuilder="dom")`	one typed tree, no treebuilder choice
`html5lib.parseFragment()`	`turbohtml.parse_fragment()`
the html5lib tokenizer	`turbohtml.tokenize()`, `turbohtml.Tokenizer`
`html5lib.serialize(tree)`	`serialize()`, `html`
`el.tag` namespaced (`{http://www.w3.org/1999/xhtml}div`)	`tag` plus `Namespace` on `namespace`
the treebuilder’s own walk and `el.attrib`	`children`, `find()` / `select()`, `attrs`

Because turbohtml returns a queryable tree, the walk-the-etree step after parsing collapses into a find or select call:

doc = parse('<ul><li class="x">a</li><li>b</li></ul>')
print([li.text for li in doc.find_all("li")])
print(doc.select_one("li.x").text)

['a', 'b']
a

Gotchas and pitfalls¶

One tree, no treebuilder. html5lib’s output shape depends on the treebuilder you pass; turbohtml has one sealed typed tree, so the node types are fixed and pattern-matchable and there is nothing to select.
Tag names are plain, namespaces are separate. html5lib’s etree treebuilder namespaces element names in Clark notation ({http://www.w3.org/1999/xhtml}div, and default namespaceHTMLElements=True). turbohtml keeps tag plain and carries the namespace on namespace:
```
from turbohtml import Namespace

svg = parse("<svg><rect/></svg>").find("rect")
print((svg.tag, svg.namespace is Namespace.SVG))
```
```
('rect', True)
```
Attributes read through ``attrs``, not ``attrib``. html5lib’s etree tree exposes el.attrib; turbohtml uses attrs, and multi-valued attributes (class, rel, …) read back as a list[str].
Encoding sniffing stops at the markup. parse runs the WHATWG byte path — BOM, then a <meta charset> prescan, then a windows-1252 fallback. html5lib with chardet installed can additionally guess from byte frequency; where that matters, detect the encoding first and hand turbohtml the decoded str.
No serializer object or filter chain. html5lib builds a serializer and threads filters through it; turbohtml serializes directly with serialize() and a Formatter, and does not offer optional-tag omission or attribute reordering.