Why a C core

Speed comes first in turbohtml, ahead of ease of maintenance. The tokenizer, the WHATWG tree builder, the CSS and XPath engines, escaping, and serialization run in C over a single arena that holds no Python objects; Python appears only at the typed edge, where a thin facade wraps the nodes you touch. This page explains what the C core buys and how it is laid out, then works through escaping and unescaping as the concrete case.

One arena per parse

A parse allocates its whole tree from one bump-allocated arena: a growing block of memory the builder hands out by advancing a pointer, never freeing an individual node. Every element, attribute, and run of text lives in that block as a compact C record with integer offsets into the source, not as a Python object. The design follows the lexbor and html5ever layouts turbohtml benchmarks against. Two properties fall out of it. Allocation is close to free, so building a tree of a hundred thousand nodes costs one arena walk rather than a hundred thousand malloc calls. And the tree frees in one step when the arena drops, so there is no per-node reference counting on the hot path.

The Python objects you handle are wrappers created lazily: a turbohtml.Node materializes when you reach a node, and it forwards to the arena record underneath. A tree you never walk into never allocates a wrapper. A read snapshots the arena before it hands control to any Python callback, which is what lets a walk stay consistent under concurrent edits (see Free-threading).

Subsystem map

The C sources under src/turbohtml/_c split by subsystem so each reads on its own, one concern at a time:

  • core – the arena, the atom table that interns tag and attribute names, and the shared string and vector primitives the rest build on.

  • tokenizer – the WHATWG tokenization state machine.

  • dom – the tree builder and the node model every query and edit runs against.

  • query – the CSS selector matcher, the find filter grammar, and the XPath 1.0 engine.

  • css – the CSS parser and the cascade that resolves a computed style.

  • js – the JavaScript tokenizer and minifier.

  • clean – the allowlist sanitizer, the autolinker, and the HTML minifier.

  • extract – main-content isolation, table reading, and structured-data collection.

  • serialize – HTML, Markdown, and plain-text output, plus canonicalization.

  • encoding – the byte sniffer and the incremental decoder.

  • unicode – normalization and the case-folding tables.

  • url – URL parsing and resolution.

  • validate – XSD, RELAX NG, and the HTML5 conformance rules.

  • data – the generated lookup tables (entity names, tag atoms, the public-suffix and IDNA data).

Escaping and unescaping sit on hot paths: HTML output escaping runs on every rendered fragment, and unescaping runs on every chunk of text an HTML parser emits. turbohtml implements both in C so they run several times faster than an equivalent pure-Python implementation, with no change in behavior.

The Performance page measures both against html.escape() and html.unescape() over real corpora. escape gains the most on text that needs little escaping (the SIMD scan classifies sixteen bytes at a time and copies clean stretches in bulk); unescape gains the most on entity-heavy input, where the standard library pays a Python call per match. The gap is narrowest on tiny strings, where call overhead dominates, and on special-dense markup, where both sides spend their time writing replacements.

Unlike a standard-library accelerator, turbohtml ships only the compiled implementation. PEP 399 requires a pure-Python fallback only for the standard library; as a third-party package distributing per-interpreter wheels, turbohtml has no need for one, which keeps the surface small.

Block-at-a-time scanning

escape spends most of its time confirming that a string contains nothing that needs escaping. For one-byte strings it classifies sixteen bytes at a time with SIMD (on arm64 NEON a single low-nibble table lookup plus one comparison matches all five specials at once; on x86-64 SSE2 compares per special; elsewhere a 64-bit SWAR word applies the has-zero bit trick). The sizing pass turns each comparison into that special’s output growth and sums the block without branches; the writing pass converts the comparisons into a position bitmask, copies clean stretches in bulk, and rewrites only the matched bytes. When nothing needs escaping, escape returns the input unchanged. Wider strings (UCS-2 / UCS-4; see PEP 393 for CPython’s string representations) pack four / two code points into a 64-bit SWAR word and probe all five special characters in a single pass. unescape works the same way in reverse: it hops between & occurrences (memchr on one-byte text) and bulk-copies the clean spans between references instead of inspecting every character. This needs the PyUnicode buffer API, which is why turbohtml cannot use the Limited API.