Tokenizer

The low-level WHATWG tokenization surface, below tree construction. tokenize() runs a whole string at once; Tokenizer streams chunks. Both yield Token objects, whose TokenType selects which attributes are meaningful.

turbohtml.tokenize(s: str, /, *, resolve_references: bool = ..., capture_source: bool = ...) → Iterator[Token]

Tokenize a whole HTML string following the WHATWG tokenization algorithm.

Parameters:

• s – the HTML to tokenize.

• resolve_references – fold character references into the surrounding text run; when False each one becomes its own CHARACTER_REFERENCE token.

• capture_source – record each markup token’s verbatim source on Token.source.

Returns:

an iterator of Token objects in document order.

Raises:

TypeError – if s is not a str.

class turbohtml.Tokenizer(*, resolve_references: bool = ..., capture_source: bool = ...)

Streaming HTML tokenizer. Feed markup with feed() and iterate the returned iterators; call close() at the end, or use the tokenizer as a context manager so leaving the with block signals end of input, then iterate the tokenizer itself for the remaining tokens. For a whole string at once use tokenize().

Parameters:

• resolve_references – fold character references into the surrounding text run; when False each one is emitted as its own CHARACTER_REFERENCE token (its data the resolved value, its source the verbatim reference). Attribute-value references are always resolved.

• capture_source – record each markup token’s verbatim source slice, available as Token.source.

close() → Iterator[Token]

Signal end of input, flushing any buffered text and the token in progress.

Returns:

an iterator over the final tokens.

feed(data: str) → Iterator[Token]

Append a chunk of markup. Text before an unfinished tag stays buffered until more is fed or close() is called.

Parameters:

data – the next chunk of markup.

Returns:

an iterator over the tokens that are now complete.

Raises:

• TypeError – if data is not a str.

• ValueError – if the tokenizer is closed; call reset() to reuse it.

reset() → None

Discard all input and return to the initial Data state.

class turbohtml.Token

An HTML token produced by Tokenizer or tokenize(). Immutable; the meaningful attributes depend on .type.

attr(name: str, default: str | None = None) → str | None

Read one attribute of a start or end tag. A valueless attribute yields the empty string.

Parameters:

• name – the attribute name.

• default – value returned when the attribute is absent.

Returns:

the attribute value, or default when it is absent.

attrs: list[tuple[str, str]] | None

attribute (name, value) pairs for tags, else None

col: int

0-based source column where this token began

data: str | None

text run, comment, or resolved character-reference data, else None

force_quirks: bool

whether a DOCTYPE forces quirks mode

line: int

1-based source line where this token began

name: str | None

DOCTYPE name, else None

public_id: str | None

DOCTYPE public identifier, else None

self_closing: bool

whether a start tag carried a trailing slash

source: str | None

verbatim source slice this token came from, else None

system_id: str | None

DOCTYPE system identifier, else None

tag: str | None

lowercased tag name for start/end tags, else None

type: TokenType

the TokenType of this token

class turbohtml.TokenType(*values)

The kind of a Token, reported by Token.type; it selects which of the token’s attributes are meaningful.

TEXT is a run of character data (Token.data); START_TAG and END_TAG are tags (Token.tag, Token.attrs); COMMENT is a comment (Token.data); DOCTYPE is a document type declaration (Token.name and the public/system ids); CHARACTER_REFERENCE is a single resolved reference emitted on its own when the tokenizer runs with resolve_references=False.

TEXT = 0

START_TAG = 1

END_TAG = 2

COMMENT = 3

DOCTYPE = 4

CHARACTER_REFERENCE = 5

SAX

Event-driven parsing that builds no tree. sax_parse() drives the WHATWG tree builder and calls a SaxHandler method for each construct; iter_events() yields the same stream as typed records. The events reflect the constructed tree (implied tags, foster parenting, the adoption agency), not the raw token stream, and no node object is created. See The event-driven parse for the memory model and Parse with SAX callbacks for a worked recipe.

turbohtml.saxparse.sax_parse(html, handler)

Parse html, calling handler’s methods for each event, and retain no tree.

Parameters:

• html (str) – the markup to parse.

• handler (SaxHandler) – a SaxHandler (or subclass) whose overridden methods receive the events.

Raises:

TypeError – if html is not a str.

Return type:

None

turbohtml.saxparse.iter_events(html)

Parse html and yield its SAX events as typed records, one at a time, without building a document.

Parameters:

html (str) – the markup to parse.

Return type:

Iterator[StartElement | EndElement | Characters | Comment | Doctype | ProcessingInstruction]

Returns:

an iterator of StartElement/EndElement/Characters/Comment/ Doctype/ProcessingInstruction records in document order.

Raises:

TypeError – if html is not a str.

class turbohtml.saxparse.SaxHandler

Base class for a sax_parse() callback object.

Every method is a no-op by default; subclass and override only the events you need, the way you subclass html.parser.HTMLParser.

start_element(tag, attrs)

Handle a start tag tag with its attrs (each a (name, value) pair, value None if valueless).

Parameters:

• tag (str)

• attrs (tuple[tuple[str, str | None], ...])

Return type:

None

end_element(tag)

Handle the end of element tag (fired for every element, including empty and void ones).

Parameters:

tag (str)

Return type:

None

characters(data)

Handle a run of character data data.

Parameters:

data (str)

Return type:

None

comment(data)

Handle a comment with body data.

Parameters:

data (str)

Return type:

None

doctype(name, public_id, system_id)

Handle the document type declaration; public_id/system_id are None when the source omits them.

Parameters:

• name (str)

• public_id (str | None)

• system_id (str | None)

Return type:

None

processing_instruction(data)

Handle a <?...> construct with body data (a WHATWG bogus comment).

Parameters:

data (str)

Return type:

None

The records iter_events() yields, unioned as SaxEvent:

class turbohtml.saxparse.StartElement(tag: str, attrs: tuple[tuple[str, str | None], ...])

A start-tag event.

Parameters:

• tag (str)

• attrs (tuple[tuple[str, str | None], ...])

tag: str

the element’s tag name.

attrs: tuple[tuple[str, str | None], ...]

the attributes, each a (name, value) pair with value None for a valueless attribute.

class turbohtml.saxparse.EndElement(tag: str)

An end-tag event, fired for every element the parser closes.

Parameters:

tag (str)

tag: str

the element’s tag name.

class turbohtml.saxparse.Characters(data: str)

A run of character data.

Parameters:

data (str)

data: str

the text.

class turbohtml.saxparse.Comment(data: str)

A comment event.

Parameters:

data (str)

data: str

the comment body, between <!-- and -->.

class turbohtml.saxparse.Doctype(name: str, public_id: str | None, system_id: str | None)

A document type declaration event.

Parameters:

• name (str)

• public_id (str | None)

• system_id (str | None)

name: str

the document type name, e.g. html.

public_id: str | None

the public identifier, or None when the source omits it.

system_id: str | None

the system identifier, or None when the source omits it.

class turbohtml.saxparse.ProcessingInstruction(data: str)

A <?...> event; WHATWG parses it as a bogus comment, so data is that comment’s body.

Parameters:

data (str)

data: str

the WHATWG bogus-comment body: the source between the opening < and the closing >, so the leading ? is kept (<?php?> gives ?php?).

turbohtml.saxparse.SaxEvent = turbohtml.saxparse.StartElement | turbohtml.saxparse.EndElement | turbohtml.saxparse.Characters | turbohtml.saxparse.Comment | turbohtml.saxparse.Doctype | turbohtml.saxparse.ProcessingInstruction

The union of every event iter_events() yields.

Rewriting

Single-pass, DOM-less rewriting in the style of Cloudflare’s lol-html. rewrite() streams the tokenizer over the input, matches CSS selectors against the open-element stack, and calls a handler for each match, applying its edits to the output as it goes – memory stays O(open-element depth), never O(document). See The streaming rewrite model for the memory model and the streamable-selector constraint, and Rewrite HTML without building a tree for worked recipes.

turbohtml.rewrite.rewrite(html, *, elements=(), text=None, comments=None, doctype=None)

Rewrite html in a single streaming pass and return the transformed markup.

Parameters:

• html (str) – the markup to rewrite.

• elements (Iterable[tuple[str, ElementHandler]]) – an iterable of (selector, handler) pairs; handler is called with the matched Element for each element the streamable CSS selector matches, in document order. When several selectors match one element their handlers run in the order given.

• text (TextHandler | None) – called with each run of character data (a text-kind Element); None skips text.

• comments (CommentHandler | None) – called with each comment (a comment-kind Element); None skips comments.

• doctype (DoctypeHandler | None) – called with the document type declaration (a doctype-kind Element); None skips it.

Return type:

str

Returns:

the rewritten markup. An untouched construct is reproduced verbatim, so a rewrite that edits nothing returns the input unchanged (character references and original quoting are preserved).

Raises:

• TypeError – if html is not a str.

• SelectorSyntaxError – if a selector is malformed or uses a construct the stream cannot match (a sibling combinator, a positional or structural pseudo-class, or :has()).

class turbohtml.rewrite.Element

The node handle a rewrite handler receives. The same handle backs every handler kind; which members apply depends on kind. It is valid only for the duration of the handler call – stashing it and using it afterwards raises RuntimeError.

kind: str

The node kind: "element", "text", "comment", or "doctype".

removed: bool

Whether a handler removed or replaced this node.

tag: str

An element’s lowercased tag name.

attrs: tuple[tuple[str, str | None], ...]

An element’s attributes as (name, value) pairs, value None for a valueless attribute.

text: str

A text or comment node’s body.

name: str | None

A doctype’s name, or None.

public_id: str | None

A doctype’s public identifier, or None.

system_id: str | None

A doctype’s system identifier, or None.

get(name, default=None)

An element attribute’s value, or default when the attribute is absent.

has_attribute(name)

Whether the element carries the named attribute.

set_attribute(name, value)

Set (or add) an element attribute.

remove_attribute(name)

Remove an element attribute if present.

set_text(value)

Replace a text or comment node’s body.

before(content, *, html=False)

Insert content immediately before this node; html=True inserts raw markup, otherwise it is escaped.

after(content, *, html=False)

Insert content immediately after this node.

prepend(content, *, html=False)

Insert content as the first of an element’s inner content.

append(content, *, html=False)

Append content as the last of an element’s inner content.

set_content(content, *, html=False)

Replace an element’s inner content, keeping its tags.

replace(content, *, html=False)

Replace this node – an element with its whole subtree – with content.

remove()

Remove this node (an element removes its whole subtree).

remove_and_keep_content()

Drop an element’s own tags but keep its children.

class turbohtml.rewrite.ElementHandler(*args, **kwargs)

A callback invoked with the matched Element for each element a selector matches.

class turbohtml.rewrite.TextHandler(*args, **kwargs)

A callback invoked with each run of character data as an Element handle of text kind.

class turbohtml.rewrite.CommentHandler(*args, **kwargs)

A callback invoked with each comment as an Element handle of comment kind.

class turbohtml.rewrite.DoctypeHandler(*args, **kwargs)

A callback invoked with the document type declaration as an Element handle of doctype kind.