Parsing

Turn markup into a navigable Document. parse() handles a whole document at once; parse_fragment() parses a fragment in a context element; IncrementalParser builds a document from chunks fed over a stream. parse_xml() switches to strict XML 1.0 well-formedness for XML rather than HTML input.

Both parse() and parse_fragment() take keyword options: encoding and detect_encoding steer decoding of bytes input, strict turns the first recovered parse error into an exception, positions records each element’s source line and column, source_locations additionally records the granular SourceLocation spans (read via Node.source_location), scripting sets the WHATWG scripting flag so <noscript> parses as a raw-text element, and allow_declarative_shadow_roots honors a <template shadowrootmode> by attaching a shadow root to its parent (on by default for parse(), off for parse_fragment()). Each is described on the function below.

turbohtml.parse(markup: str | bytes, *, encoding: str | None = None, strict: bool = False, detect_encoding: bool = False, positions: bool = True, source_locations: bool = False, scripting: bool = False, allow_declarative_shadow_roots: bool = True) → Document

Parse a whole HTML document with the WHATWG tree-construction algorithm and return a navigable Document.

Parameters:

• markup – the document, as str, or bytes whose encoding is sniffed (the encoding argument, then a <meta> charset, then windows-1252).

• encoding – the encoding to decode bytes with; a declared, <meta>, or BOM encoding still wins over it.

• strict – raise the first recovered parse error as HTMLParseError instead of collecting it on Document.errors.

• detect_encoding – add a content-based detection step for bytes input, used only when the spec’s encoding steps yield nothing.

• positions – record each element’s source_line/source_col; pass False to skip it when memory or speed matters more than source locations.

• source_locations – also record each element’s granular start-/end-tag and per-attribute spans, read via Element.source_location (parse5’s sourceCodeLocationInfo). Off by default; implies positions when on.

• scripting – set the WHATWG scripting flag on. With it on, <noscript> is a raw-text element – its content is raw text, not markup, and serializes unescaped – reproducing the tree a scripting browser builds. Off by default so <noscript> content stays parsed and accessible.

• allow_declarative_shadow_roots – honor a <template shadowrootmode>, which attaches a shadow root to its parent element and parses the template’s content into it (Element.shadow_root). On by default, matching a browser navigating to the document; pass False to keep such templates as ordinary template elements.

Returns:

the parsed Document.

Raises:

• TypeError – if markup is neither a str nor a bytes-like object.

• LookupError – if encoding names a codec Python does not know.

• HTMLParseError – under strict=True, on the first recovered parse error; its error attribute carries the ParseError (code, line, col).

turbohtml.parse_xml(markup: str) → Document

Parse a whole document under XML 1.0 well-formedness rather than the WHATWG HTML rules, and return a navigable Document.

Unlike parse(), this applies XML productions with no HTML recovery: element and attribute names are case-sensitive, <x/> self-closes any element, CDATA sections and processing instructions become their own nodes, only the five predefined entities (amp, lt, gt, quot, apos) and numeric character references resolve, and a namespace prefix must be declared with xmlns. The first well-formedness violation stops the parse and raises.

Parameters:

markup – the XML document, as str.

Returns:

the parsed Document.

Raises:

• TypeError – if markup is not a str.

• HTMLParseError – on the first well-formedness error (a mismatched or unclosed tag, an undeclared namespace prefix, an undefined entity, a duplicate attribute, and the rest); its error attribute carries the ParseError (code, line, col).

turbohtml.parse_fragment(html: str, context: str = 'div', *, positions: bool = True, source_locations: bool = False, scripting: bool = False, allow_declarative_shadow_roots: bool = False) → Element

Parse an HTML fragment as the innerHTML of a context element.

Parameters:

• html – the fragment markup.

• context – the context element’s tag name, optionally namespaced (e.g. ‘td’, ‘svg path’).

• positions – record each element’s source_line/source_col; pass False to skip it.

• source_locations – also record each element’s granular start-/end-tag and per-attribute spans, read via Element.source_location. Implies positions.

• scripting – set the WHATWG scripting flag on, making <noscript> a raw-text element (see parse). Off by default.

• allow_declarative_shadow_roots – honor a <template shadowrootmode> by attaching a shadow root to its parent (see parse). Off by default, matching the innerHTML fragment case; the setHTMLUnsafe path turns it on.

Returns:

the context Element with the parsed nodes as its children.

Raises:

• TypeError – if html or context is not a str.

• ValueError – if context is not a known element tag.

class turbohtml.IncrementalParser(*, encoding: str = 'utf-8', positions: bool = True, source_locations: bool = False)

A push parser that builds a Document from chunks fed with feed(), so a document arriving over a stream never has to be held whole in memory. Feed str or bytes chunks in any size, then call close() for the finished Document. For a whole string or bytes at once use parse().

Parameters:

• encoding – codec used to decode any bytes chunks.

• positions – whether to record each element’s source line and column; pass False to skip it when memory or speed matters more.

• source_locations – whether to record each element’s granular start- and end-tag and per-attribute spans, read via Element.source_location.

close() → Document

Signal end of input, flushing the decoder and tokenizer and applying the missing html/head/body rules. Raises ValueError if the parser is already closed.

Returns:

the finished Document.

feed(data: str | bytes) → None

Push a chunk of markup, so the source need never be held whole in memory. Raises ValueError once the parser is closed.

Parameters:

data – str (fed as code points) or a bytes-like object decoded with the parser’s encoding, an incomplete trailing multi-byte sequence held back until the next chunk.

Raises:

• TypeError – if data is neither a str nor a bytes-like object.

• ValueError – if the parser is already closed.

• LookupError – if the parser’s encoding names a codec Python does not know (raised on the first bytes chunk).

class turbohtml.Document

A parsed document: the root of the tree returned by parse().

base_url(fallback: str = '') → str

Resolve this document’s base URL from its first <base href>.

Parameters:

fallback – the URL the <base href> is resolved against, and the result itself when the document has no <base href>.

Returns:

the document’s base URL.

dublin_core() → dict[str, str]

Return a dict mapping each <meta name=”dc.*”> or <meta name=”dcterms.*”> name (lower-cased) to its content value. When a name repeats, the last occurrence wins.

encoding: str | None

the resolved encoding name for bytes input, or None for str

errors: list[ParseError]

the WHATWG parse errors detected, as a list of ParseError in document order

feed() → Feed | None

Normalize an RSS 2.0, Atom 1.0, or RDF/RSS-1.0 document into one Feed record, or None when the document carries no feed root. Feed has .type (“rss”, “atom”, or “rdf”), .title, .link, .description, .updated, and .entries (a tuple of Entry). Each Entry has .title, .link, .id, .updated, .published, .summary, .content, and .author, each the first present value across the RSS/Atom/RDF spellings of the field.

json_ld() → list[JSONValue]

Parse every <script type=”application/ld+json”> block in the document with the standard library json module, returning the list of decoded values in document order. A block that is not valid JSON is skipped.

meta_refresh(fallback: str = '') → tuple[float, str] | None

Read the document’s <meta http-equiv=refresh> redirect. A refresh tag inside <noscript> is ignored.

Parameters:

fallback – the URL the directive’s target is resolved against, and the target itself when the directive omits one.

Returns:

a (delay_seconds, url) pair, or None when the document has no refresh directive.

microdata(base_url: str | None = None) → list[MicrodataItem]

Extract HTML Microdata as a list of MicrodataItem, one per top-level itemscope. Each item has .properties (a dict mapping each itemprop name to its list of values), plus .type and .id carrying the itemtype/itemid attribute or None. A property value is a nested MicrodataItem for an itemscope, otherwise the element’s microdata value.

Parameters:

base_url – when given, the URL each relative URL-valued property (an a/area/link href, a media src, an object data) is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

opengraph(base_url: str | None = None) → OpenGraph

Return an OpenGraph record of the page’s Open Graph metadata, a successor to the opengraph library. Each <meta property=”og:…”> key is read with the og: prefix stripped (og:title reads as og[“title”]) and the twitter: keys are dropped; when a key repeats, the last occurrence wins.

Parameters:

base_url – when given, the URL each relative URL-valued key (og:url, og:image, og:video, …) is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

rdfa(base_url: str | None = None) → list[RdfaItem]

Extract RDFa as a list of RdfaItem, one per top-level typeof resource. Each item has .properties (a dict mapping each expanded property IRI to its list of values), plus .type (the expanded typeof IRIs), .resource (the subject), and .vocab (the in-scope @vocab). property/typeof tokens expand against @vocab and @prefix (the RDFa 1.1 initial context seeds the common prefixes); an undeclared prefix stays verbatim.

Parameters:

base_url – when given, the URL each resource/href/src IRI is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

root: Element | None

the root <html> element, or None

structured_data(base_url: str | None = None) → StructuredData

Extract every supported structured-data format from the document, a successor to extruct. Returns a StructuredData record with .json_ld (list of parsed JSON-LD values), .microdata (list of MicrodataItem), .opengraph (dict of og:/twitter: keys to their content), .rdfa (list of RdfaItem), .dublin_core (dict of dc.*/dcterms.* names to their content), and .microformats (an empty list, a later phase).

Parameters:

base_url – when given, the URL each relative URL-valued microdata, opengraph, or RDFa field is resolved against (a <base href> refines it, HTML spec 4.2.3); json_ld and Dublin Core are left verbatim. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

class turbohtml.ParseError

One WHATWG parse error collected during parse(): a code and a source position.

code is the spec error code, line is 1-based and col is 0-based (as for Token).

code: str

the WHATWG parse-error code, e.g. ‘unexpected-question-mark-instead-of-tag-name’

col: int

the 0-based source column where the error was detected

line: int

the 1-based source line where the error was detected

exception turbohtml.HTMLParseError

Raised by parse(strict=True) on the first parse error; .error is the ParseError.

turbohtml.treebuild

Retarget the parser at a tree you control. parse_into() runs the same WHATWG tree builder and drives a builder object – one create_* per node kind plus an append – to construct the tree directly, with no navigable Node materialized and no second walk. It is Rust html5ever’s TreeSink and Node parse5’s TreeAdapter in turbohtml shape. See Retargeting the parser for how it differs from the SAX stream and Parse into your own tree for worked builders.

turbohtml.treebuild.parse_into(html, builder)

Parse html and drive builder to construct the tree, returning its document handle.

The document is parsed with the full WHATWG tree-construction algorithm, then its nodes are handed to builder in document order: one create_* call per node followed by an append() linking it under its parent. No navigable Node is built and the tree is walked only once.

Parameters:

• html (str) – the markup to parse.

• builder (TreeBuilder[TypeVar(H)]) – the TreeBuilder (any object with its methods) that constructs your representation.

Return type:

TypeVar(H)

Returns:

the handle create_document() returned, now populated with the parsed tree.

Raises:

• TypeError – if html is not a str.

• AttributeError – if builder is missing one of the TreeBuilder methods.

class turbohtml.treebuild.TreeBuilder(*args, **kwargs)

The tree-construction sink parse_into() drives, parse5’s TreeAdapter in turbohtml shape.

Implement each method to create the matching node in your own representation and return a handle for it; the handle is opaque to turbohtml, threaded back only into append(). The protocol is structural, so any object with these methods works – no base class to inherit.

create_document()

Create the document root and return its handle; parse_into() returns this handle.

Return type:

TypeVar(H)

create_doctype(name, public_id, system_id)

Create a doctype node; public_id/system_id are None when the source omits them.

Parameters:

• name (str)

• public_id (str | None)

• system_id (str | None)

Return type:

TypeVar(H)

create_element(name, namespace, attrs)

Create an element name in namespace (a URI) with attrs (each a (name, value) pair).

Parameters:

• name (str)

• namespace (str)

• attrs (tuple[tuple[str, str | None], ...])

Return type:

TypeVar(H)

create_text(data)

Create a text node holding data.

Parameters:

data (str)

Return type:

TypeVar(H)

create_comment(data)

Create a comment node holding data.

Parameters:

data (str)

Return type:

TypeVar(H)

create_pi(data)

Create a node for a <?...> construct (a WHATWG bogus comment) holding data.

Parameters:

data (str)

Return type:

TypeVar(H)

append(parent, child)

Append the child handle under the parent handle, in document order.

Parameters:

• parent (TypeVar(H))

• child (TypeVar(H))

Return type:

None

turbohtml.migration.stdlib

A drop-in base class for html.parser.HTMLParser subclasses, over turbohtml’s WHATWG-conformant tokenizer. Subclass it, override the handle_* methods, and feed input incrementally as with the standard library.

class turbohtml.migration.stdlib.HTMLParser(*, convert_charrefs=True)

A subclass-and-override callback parser over the streaming tokenizer.

Parameters:

convert_charrefs (bool) – accepted for compatibility but ignored; references are always resolved.

convert_charrefs: bool

Accepted for compatibility; references are always resolved regardless of its value.

feed(data)

Feed a chunk of HTML, dispatching the tokens it completes to the handlers.

Parameters:

data (str) – the next chunk of HTML.

Return type:

None

close()

Signal end of input, dispatching any tokens still buffered.

Return type:

None

reset()

Discard the buffered input and position, ready to parse a new document.

Return type:

None

getpos()

Report the source position of the last dispatched token.

Return type:

tuple[int, int]

Returns:

the 1-based line and 0-based column of the token last dispatched.

handle_starttag(tag, attrs)

Handle a start tag; override to act on it.

Parameters:

• tag (str) – the lowercased tag name.

• attrs (list[tuple[str, str | None]]) – the (name, value) attribute pairs, a valueless attribute’s value the empty string.

Return type:

None

handle_startendtag(tag, attrs)

Handle a self-closing tag; by default fire start then end, like the standard library.

Parameters:

• tag (str) – the lowercased tag name.

• attrs (list[tuple[str, str | None]]) – the (name, value) attribute pairs.

Return type:

None

handle_endtag(tag)

Handle an end tag; override to act on it.

Parameters:

tag (str) – the lowercased tag name.

Return type:

None

handle_data(data)

Handle a run of text, with character references resolved.

Parameters:

data (str) – the decoded text.

Return type:

None

handle_comment(data)

Handle a comment (also processing instructions and CDATA, which are comments here).

Parameters:

data (str) – the comment text, without the <!-- --> delimiters.

Return type:

None

handle_decl(decl)

Handle a <!DOCTYPE ...> declaration, with the leading <! and trailing > removed.

Parameters:

decl (str) – the declaration text, such as DOCTYPE html.

Return type:

None

handle_pi(data)

Handle a processing instruction; never called, since PIs are comments in the HTML spec.

Parameters:

data (str) – the instruction text.

Return type:

None

handle_entityref(name)

Handle a named reference; never called, since references are always resolved.

Parameters:

name (str) – the entity name, without the & and ;.

Return type:

None

handle_charref(name)

Handle a numeric reference; never called, since references are always resolved.

Parameters:

name (str) – the numeric reference body, without the &# and ;.

Return type:

None

unknown_decl(data)

Handle a CDATA section; never called, since it is a comment outside foreign content.

Parameters:

data (str) – the section text.

Return type:

None