Tokenizer

The low-level WHATWG tokenization surface, below tree construction. tokenize() runs a whole string at once; Tokenizer streams chunks. Both yield Token objects, whose TokenType selects which attributes are meaningful.

turbohtml.tokenize(s: str, /, *, resolve_references: bool = ..., capture_source: bool = ...) → Iterator[Token]

Tokenize a whole HTML string following the WHATWG tokenization algorithm.

Parameters:

• s – the HTML to tokenize.

• resolve_references – fold character references into the surrounding text run; when False each one becomes its own CHARACTER_REFERENCE token.

• capture_source – record each markup token’s verbatim source on Token.source.

Returns:

an iterator of Token objects in document order.

Raises:

TypeError – if s is not a str.

class turbohtml.Tokenizer(*, resolve_references: bool = ..., capture_source: bool = ...)

Streaming HTML tokenizer. Feed markup with feed() and iterate the returned iterators; call close() at the end, or use the tokenizer as a context manager so leaving the with block signals end of input, then iterate the tokenizer itself for the remaining tokens. For a whole string at once use tokenize().

Parameters:

• resolve_references – fold character references into the surrounding text run; when False each one is emitted as its own CHARACTER_REFERENCE token (its data the resolved value, its source the verbatim reference). Attribute-value references are always resolved.

• capture_source – record each markup token’s verbatim source slice, available as Token.source.

close() → Iterator[Token]

Signal end of input, flushing any buffered text and the token in progress.

Returns:

an iterator over the final tokens.

feed(data: str) → Iterator[Token]

Append a chunk of markup. Text before an unfinished tag stays buffered until more is fed or close() is called.

Parameters:

data – the next chunk of markup.

Returns:

an iterator over the tokens that are now complete.

Raises:

• TypeError – if data is not a str.

• ValueError – if the tokenizer is closed; call reset() to reuse it.

reset() → None

Discard all input and return to the initial Data state.

class turbohtml.Token

An HTML token produced by Tokenizer or tokenize(). Immutable; the meaningful attributes depend on .type.

attr(name: str, default: str | None = None) → str | None

Read one attribute of a start or end tag. A valueless attribute yields the empty string.

Parameters:

• name – the attribute name.

• default – value returned when the attribute is absent.

Returns:

the attribute value, or default when it is absent.

attrs: list[tuple[str, str]] | None

attribute (name, value) pairs for tags, else None

col: int

0-based source column where this token began

data: str | None

text run, comment, or resolved character-reference data, else None

force_quirks: bool

whether a DOCTYPE forces quirks mode

line: int

1-based source line where this token began

name: str | None

DOCTYPE name, else None

public_id: str | None

DOCTYPE public identifier, else None

self_closing: bool

whether a start tag carried a trailing slash

source: str | None

verbatim source slice this token came from, else None

system_id: str | None

DOCTYPE system identifier, else None

tag: str | None

lowercased tag name for start/end tags, else None

type: TokenType

the TokenType of this token

class turbohtml.TokenType(*values)

The kind of a Token, reported by Token.type; it selects which of the token’s attributes are meaningful.

TEXT is a run of character data (Token.data); START_TAG and END_TAG are tags (Token.tag, Token.attrs); COMMENT is a comment (Token.data); DOCTYPE is a document type declaration (Token.name and the public/system ids); CHARACTER_REFERENCE is a single resolved reference emitted on its own when the tokenizer runs with resolve_references=False.

TEXT = 0

START_TAG = 1

END_TAG = 2

COMMENT = 3

DOCTYPE = 4

CHARACTER_REFERENCE = 5