From the standard library¶

Python’s standard library ships HTML primitives in the html package. html.escape() and html.unescape() handle entity encoding and decoding, html.entities exposes the reference tables, and html.parser.HTMLParser is a SAX-style tokenizer you subclass and drive with handle_* callbacks. These are the zero-dependency, always-available building blocks that ship with CPython; many scripts, templating helpers, and scrapers reach for them because they are already installed. The scope stops at tokenizing and entity work: html.parser does not build a document tree, does not implement WHATWG error recovery, and is explicitly documented as not fully HTML5-conformant.

turbohtml covers that same ground and extends past it. turbohtml.escape() and turbohtml.unescape() match the stdlib functions byte for byte, turbohtml.tokenize() and turbohtml.Tokenizer replace the callback tokenizer, and turbohtml.migration.stdlib.HTMLParser keeps your existing handle_* subclass working unchanged. Everything runs over a WHATWG-conformant C core that also builds a full parse tree, which html.parser has no equivalent for. The same C core also covers unicodedata’s Unicode normalization, so turbohtml.detect.normalize() is a drop-in for unicodedata.normalize().

turbohtml vs stdlib¶

Dimension	turbohtml	stdlib (`html`)
Scope	Escape/unescape, tokenizer, and full WHATWG tree construction	Escape/unescape, entity tables, and a tokenizer only (no tree)
Feature breadth	Tokens, tree, selectors, serialization, plus the callback shim	`escape`/`unescape`, `html.entities` tables, `HTMLParser` callbacks
Performance	SIMD scanning, several times faster on escape/unescape	Pure-Python entity scan and tokenizer
Typing	Fully type annotated across the public surface	Annotated in typeshed stubs, not conformant behavior
Dependencies	Compiled C extension (wheels), installed from PyPI	Built into CPython, zero install
Maintenance	Actively developed, tracks the WHATWG spec	Stable CPython module, `html.parser` frozen as non-conformant

Feature overlap¶

The shared surface ports one-to-one:

html.escape() → turbohtml.escape(), same signature and output.
html.unescape() → turbohtml.unescape(), same signature and output.
html.parser.HTMLParser subclasses → turbohtml.migration.stdlib.HTMLParser, same handle_* callbacks and feed/close/reset/getpos methods.

What turbohtml adds¶

WHATWG-conformant tokenizing and tree construction via turbohtml.parse() and turbohtml.parse_fragment(); html.parser tokenizes but never builds a tree and is documented as not HTML5-conformant.
A token stream you drive yourself through turbohtml.tokenize() and turbohtml.Tokenizer, instead of inverting control into callbacks.
An event-driven parse, turbohtml.saxparse.sax_parse(), that keeps html.parser’s callback shape but fires on the constructed tree – implied tags, foster parenting, the adoption agency – which the standard library never builds.
Verbatim source capture per token (capture_source=True → token.source) and unresolved reference tokens (resolve_references=False → TokenType.CHARACTER_REFERENCE).
SIMD-accelerated escape/unescape scanning.

What stdlib has that turbohtml does not¶

html.parser and the html functions are built into CPython with no install step. turbohtml ships a compiled extension from PyPI; in environments that cannot install wheels or build C, the stdlib remains the only option.
html.entities exposes the raw reference tables (name2codepoint, codepoint2name, html5) as public data. turbohtml resolves references through escape/unescape and the tokenizer rather than exposing the dicts; if you consume those tables directly, keep importing html.entities.

Performance¶

operation	turbohtml	stdlib
escape — tiny plain (64 B)	54.5 ns	118 ns (2.2x)
escape — medium markup (4 KiB)	2.2 µs	7.48 µs (3.4x)
escape — no-op prose (4 MiB)	115 µs	2.56 ms (22.3x)
escape — book text (3 MiB)	669 µs	2.65 ms (4.0x)
escape — book HTML (4 MiB)	1.25 ms	4.65 ms (3.8x)
escape — spec HTML, dense (4 MiB)	4.94 ms	12.8 ms (2.6x)
escape — UCS-2 plain (4 MiB)	794 µs	2.46 ms (3.1x)
escape — UCS-2 markup (4 MiB)	5.62 ms	11 ms (2.0x)
escape — UCS-4 plain (4 MiB)	918 µs	5.29 ms (5.8x)
escape — UCS-4 markup (4 MiB)	7.2 ms	19.4 ms (2.7x)
unescape — tiny plain (64 B)	33.3 ns	39 ns (1.2x)
unescape — medium dense refs (4 KiB)	7 µs	71.4 µs (10.2x)
unescape — numeric refs (4 KiB)	4.84 µs	79.1 µs (16.4x)
unescape — book HTML, real refs (4 MiB)	3.29 ms	8.32 ms (2.6x)
unescape — escaped book HTML (5 MiB)	1.61 ms	19.8 ms (12.4x)
unescape — dense refs (4 MiB)	8.29 ms	73.6 ms (8.9x)
unescape — UCS-2 refs (4 MiB)	2.47 ms	18.6 ms (7.6x)
tokenize — typical markup	29.2 µs	437 µs (15.0x)
tokenize — text-heavy prose	543 ns	2.78 µs (5.2x)
tokenize — attribute-heavy	18.2 µs	305 µs (16.8x)
tokenize — script-heavy	11.2 µs	153 µs (13.7x)
tokenize — entity-heavy	20 µs	191 µs (9.6x)
tokenize — wpt tiny (0.6 kB)	1.5 µs	17.2 µs (11.5x)
tokenize — wpt small (4 kB)	12 µs	168 µs (14.1x)
tokenize — wpt medium (9.6 kB)	27.5 µs	360 µs (13.1x)
tokenize — wpt large (92 kB)	322 µs	3.99 ms (12.5x)
tokenize — wpt CJK (124 kB)	548 µs	8.49 ms (15.6x)
tokenize — whatwg spec (235 kB)	767 µs	7.65 ms (10.0x)
tokenize — ecmascript spec (3 MB)	6.2 ms	54 ms (8.8x)
tokenize — whatwg spec source (7.9 MB)	36.3 ms	384 ms (10.6x)
feed and dispatch a page — daring fireball (10 kB)	86.9 µs	308 µs (3.6x)
feed and dispatch a page — ars technica (56 kB)	374 µs	1.39 ms (3.8x)
feed and dispatch a page — mozilla blog (95 kB)	801 µs	3.18 ms (4.0x)
feed and dispatch a page — whatwg spec (235 kB)	2.5 ms	7.61 ms (3.1x)

turbohtml.escape() and turbohtml.unescape() reproduce the standard-library functions byte for byte, so they are drop-ins, but scan with SIMD and run several times faster.

How to migrate¶

Swap the imports and, if you subclass the parser, swap the base class:

stdlib call	turbohtml call
`html.escape(s)`	`turbohtml.escape(s)`
`html.unescape(s)`	`turbohtml.unescape(s)`
`class P(html.parser.HTMLParser)`	`class P(turbohtml.migration.stdlib.HTMLParser)`
`handle_starttag(tag, attrs)`	`token.type is TokenType.START_TAG` → `token.tag`, `token.attrs`
`handle_startendtag(tag, attrs)`	`TokenType.START_TAG` with `token.self_closing`
`handle_endtag(tag)`	`TokenType.END_TAG` → `token.tag`
`handle_data(data)`	`TokenType.TEXT` → `token.data`
`handle_comment(data)`	`TokenType.COMMENT` → `token.data`
`handle_decl(decl)`	`TokenType.DOCTYPE` → `token.name`
`handle_entityref`/`handle_charref`	`tokenize(..., resolve_references=False)` → `TokenType.CHARACTER_REFERENCE`, else resolved in `token.data`
`get_starttag_text()`	`tokenize(..., capture_source=True)` → `token.source`

Escape and unescape are literal drop-ins:

import html
from turbohtml import escape, unescape

print(escape('<a href="x">') == html.escape('<a href="x">'))
print(unescape("caf&eacute; &#127881;") == html.unescape("caf&eacute; &#127881;"))

True
True

To keep an existing html.parser.HTMLParser subclass, swap its base class for turbohtml.migration.stdlib.HTMLParser: the same handle_* callbacks and feed/close methods run over the WHATWG-conformant tokenizer. Or drop the subclass and take the token stream from turbohtml.tokenize() (or turbohtml.Tokenizer.feed() for incremental input), or skip tokens entirely and turbohtml.parse() straight to a tree. All three are WHATWG-conformant, unlike html.parser. The How-to guides guide has a worked port.

HTMLParser is a SAX-style callback API; turbohtml gives you the events as a token stream you drive yourself, which inverts the control flow. Each handle_* override becomes a branch on Token.type:

import turbohtml
from turbohtml import TokenType

events = []
for token in turbohtml.tokenize('<p class="x">Hi &amp; bye</p>'):
    if token.type is TokenType.START_TAG:
        events.append(("start", token.tag, token.attrs))
    elif token.type is TokenType.TEXT:
        events.append(("data", token.data))
    elif token.type is TokenType.END_TAG:
        events.append(("end", token.tag))
print(events)

[('start', 'p', [('class', 'x')]), ('data', 'Hi & bye'), ('end', 'p')]

If you liked html.parser’s callback shape and only want the WHATWG-correct tree behind it, turbohtml.saxparse.sax_parse() keeps that shape: subclass turbohtml.saxparse.SaxHandler, override the events you need, and the parser fires them on the constructed tree rather than the raw tags. The handle_* methods map onto SAX methods one-to-one:

`html.parser` override	`SaxHandler` override
`handle_starttag(tag, attrs)`	`start_element(tag, attrs)`
`handle_endtag(tag)`	`end_element(tag)`
`handle_data(data)`	`characters(data)`
`handle_comment(data)`	`comment(data)`
`handle_decl(decl)`	`doctype(name, public_id, system_id)`
`handle_pi(data)`	`processing_instruction(data)`

from turbohtml.saxparse import SaxHandler, sax_parse


class Collector(SaxHandler):
    def __init__(self):
        self.starts = []

    def start_element(self, tag, attrs):
        self.starts.append(tag if not attrs else f"{tag} {dict(attrs)}")


collector = Collector()
sax_parse("<table><td>cell", collector)
print(collector.starts)

['html', 'head', 'body', 'table', 'tbody', 'tr', 'td']

html.parser would report just table and td; the SAX events carry the implied html/head/body and the foster-parented tbody/tr the tree builder inserts. Unlike html.parser, sax_parse builds the working tree (freed at the end) rather than streaming in constant space, so it is the tool for a spec-correct one-pass extraction, not for a document larger than memory. The Parse with SAX callbacks guide has more, and The event-driven parse covers the memory model.

Unicode normalization¶

unicodedata.normalize() and unicodedata.is_normalized() move to turbohtml.detect.normalize() and turbohtml.detect.is_normalized(): the form name comes first and the output is identical, because turbohtml runs the four forms in C over tables generated from the interpreter’s own unicodedata. A quick check returns already-normalized text untouched.

stdlib call	turbohtml call
`unicodedata.normalize("NFC", s)`	`turbohtml.detect.normalize("NFC", s)`
`unicodedata.is_normalized("NFC", s)`	`turbohtml.detect.is_normalized("NFC", s)`

import unicodedata

from turbohtml.detect import normalize

forms = ("NFC", "NFD", "NFKC", "NFKD")
text = "ﬁ café ẛ̣"
print(all(normalize(form, text) == unicodedata.normalize(form, text) for form in forms))

True

Gotchas and pitfalls¶

The token stream inverts html.parser’s callback control flow: you loop over tokens and branch on Token.type instead of overriding handle_* (unless you subclass turbohtml.migration.stdlib.HTMLParser, which keeps the callbacks).
By default token.data already holds decoded text (the equivalent of convert_charrefs=True). To recover the split stream convert_charrefs=False gives, pass resolve_references=False and handle TokenType.CHARACTER_REFERENCE tokens, whose token.source is the verbatim reference and token.data its resolved value. On turbohtml.migration.stdlib.HTMLParser the convert_charrefs argument is accepted for signature compatibility but ignored; references are always resolved.
The verbatim start-tag text get_starttag_text() returns is token.source once you pass capture_source=True; it is not captured by default.
html.parser is documented as not fully HTML5-conformant, so tricky recovery cases (malformed tags, misnested elements, foreign content) can tokenize differently. turbohtml follows the WHATWG spec, so output may diverge from a legacy html.parser run on the same broken input; the turbohtml result is the conformant one.
If your code imports the reference tables from html.entities directly, keep that import: turbohtml does not re-export name2codepoint/codepoint2name/html5.