Parsing

Turn markup into a navigable Document. parse() handles a whole document at once; parse_fragment() parses a fragment in a context element; IncrementalParser builds a document from chunks fed over a stream.

turbohtml.parse(markup: str | bytes, *, encoding: str | None = None, strict: bool = False, detect_encoding: bool = False, positions: bool = True) → Document

Parse a whole HTML document with the WHATWG tree-construction algorithm and return a navigable Document.

Parameters:

• markup – the document, as str, or bytes whose encoding is sniffed (the encoding argument, then a <meta> charset, then windows-1252).

• encoding – the encoding to decode bytes with; a declared, <meta>, or BOM encoding still wins over it.

• strict – raise the first recovered parse error as HTMLParseError instead of collecting it on Document.errors.

• detect_encoding – add a content-based detection step for bytes input, used only when the spec’s encoding steps yield nothing.

• positions – record each element’s source_line/source_col; pass False to skip it when memory or speed matters more than source locations.

Returns:

the parsed Document.

Raises:

• TypeError – if markup is neither a str nor a bytes-like object.

• LookupError – if encoding names a codec Python does not know.

• HTMLParseError – under strict=True, on the first recovered parse error; its error attribute carries the ParseError (code, line, col).

turbohtml.parse_fragment(html: str, context: str = 'div', *, positions: bool = True) → Element

Parse an HTML fragment as the innerHTML of a context element.

Parameters:

• html – the fragment markup.

• context – the context element’s tag name, optionally namespaced (e.g. ‘td’, ‘svg path’).

• positions – record each element’s source_line/source_col; pass False to skip it.

Returns:

the context Element with the parsed nodes as its children.

Raises:

• TypeError – if html or context is not a str.

• ValueError – if context is not a known element tag.

class turbohtml.IncrementalParser(*, encoding: str = 'utf-8', positions: bool = True)

A push parser that builds a Document from chunks fed with feed(), so a document arriving over a stream never has to be held whole in memory. Feed str or bytes chunks in any size, then call close() for the finished Document. For a whole string or bytes at once use parse().

Parameters:

• encoding – codec used to decode any bytes chunks.

• positions – whether to record each element’s source line and column; pass False to skip it when memory or speed matters more.

close() → Document

Signal end of input, flushing the decoder and tokenizer and applying the missing html/head/body rules. Raises ValueError if the parser is already closed.

Returns:

the finished Document.

feed(data: str | bytes) → None

Push a chunk of markup, so the source need never be held whole in memory. Raises ValueError once the parser is closed.

Parameters:

data – str (fed as code points) or a bytes-like object decoded with the parser’s encoding, an incomplete trailing multi-byte sequence held back until the next chunk.

Raises:

• TypeError – if data is neither a str nor a bytes-like object.

• ValueError – if the parser is already closed.

• LookupError – if the parser’s encoding names a codec Python does not know (raised on the first bytes chunk).

class turbohtml.Document

A parsed document: the root of the tree returned by parse().

base_url(fallback: str = '') → str

Resolve this document’s base URL from its first <base href>.

Parameters:

fallback – the URL the <base href> is resolved against, and the result itself when the document has no <base href>.

Returns:

the document’s base URL.

dublin_core() → dict[str, str]

Return a dict mapping each <meta name=”dc.*”> or <meta name=”dcterms.*”> name (lower-cased) to its content value. When a name repeats, the last occurrence wins.

encoding: str | None

the resolved encoding name for bytes input, or None for str

errors: list[ParseError]

the WHATWG parse errors detected, as a list of ParseError in document order

json_ld() → list[JSONValue]

Parse every <script type=”application/ld+json”> block in the document with the standard library json module, returning the list of decoded values in document order. A block that is not valid JSON is skipped.

meta_refresh(fallback: str = '') → tuple[float, str] | None

Read the document’s <meta http-equiv=refresh> redirect. A refresh tag inside <noscript> is ignored.

Parameters:

fallback – the URL the directive’s target is resolved against, and the target itself when the directive omits one.

Returns:

a (delay_seconds, url) pair, or None when the document has no refresh directive.

microdata(base_url: str | None = None) → list[MicrodataItem]

Extract HTML Microdata as a list of MicrodataItem, one per top-level itemscope. Each item has .properties (a dict mapping each itemprop name to its list of values), plus .type and .id carrying the itemtype/itemid attribute or None. A property value is a nested MicrodataItem for an itemscope, otherwise the element’s microdata value.

Parameters:

base_url – when given, the URL each relative URL-valued property (an a/area/link href, a media src, an object data) is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

opengraph(base_url: str | None = None) → OpenGraph

Return an OpenGraph record of the page’s Open Graph metadata, a successor to the opengraph library. Each <meta property=”og:…”> key is read with the og: prefix stripped (og:title reads as og[“title”]) and the twitter: keys are dropped; when a key repeats, the last occurrence wins.

Parameters:

base_url – when given, the URL each relative URL-valued key (og:url, og:image, og:video, …) is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

rdfa(base_url: str | None = None) → list[RdfaItem]

Extract RDFa as a list of RdfaItem, one per top-level typeof resource. Each item has .properties (a dict mapping each expanded property IRI to its list of values), plus .type (the expanded typeof IRIs), .resource (the subject), and .vocab (the in-scope @vocab). property/typeof tokens expand against @vocab and @prefix (the RDFa 1.1 initial context seeds the common prefixes); an undeclared prefix stays verbatim.

Parameters:

base_url – when given, the URL each resource/href/src IRI is resolved against; a <base href> refines it. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

root: Element | None

the root <html> element, or None

structured_data(base_url: str | None = None) → StructuredData

Extract every supported structured-data format from the document, a successor to extruct. Returns a StructuredData record with .json_ld (list of parsed JSON-LD values), .microdata (list of MicrodataItem), .opengraph (dict of og:/twitter: keys to their content), .rdfa (list of RdfaItem), .dublin_core (dict of dc.*/dcterms.* names to their content), and .microformats (an empty list, a later phase).

Parameters:

base_url – when given, the URL each relative URL-valued microdata, opengraph, or RDFa field is resolved against (a <base href> refines it, HTML spec 4.2.3); json_ld and Dublin Core are left verbatim. None (the default) returns every value verbatim.

Raises:

ValueError – if base_url is not a valid absolute URL.

class turbohtml.ParseError

One WHATWG parse error collected during parse(): a code and a source position.

code is the spec error code, line is 1-based and col is 0-based (as for Token).

code: str

the WHATWG parse-error code, e.g. ‘unexpected-question-mark-instead-of-tag-name’

col: int

the 0-based source column where the error was detected

line: int

the 1-based source line where the error was detected

exception turbohtml.HTMLParseError

Raised by parse(strict=True) on the first parse error; .error is the ParseError.

turbohtml.migration.stdlib

A drop-in base class for html.parser.HTMLParser subclasses, over turbohtml’s WHATWG-conformant tokenizer. Subclass it, override the handle_* methods, and feed input incrementally as with the standard library.

class turbohtml.migration.stdlib.HTMLParser(*, convert_charrefs=True)

A subclass-and-override callback parser over the streaming tokenizer.

Parameters:

convert_charrefs (bool) – accepted for compatibility but ignored; references are always resolved.

convert_charrefs: bool

Accepted for compatibility; references are always resolved regardless of its value.

feed(data)

Feed a chunk of HTML, dispatching the tokens it completes to the handlers.

Parameters:

data (str) – the next chunk of HTML.

Return type:

None

close()

Signal end of input, dispatching any tokens still buffered.

Return type:

None

reset()

Discard the buffered input and position, ready to parse a new document.

Return type:

None

getpos()

Report the source position of the last dispatched token.

Return type:

tuple[int, int]

Returns:

the 1-based line and 0-based column of the token last dispatched.

handle_starttag(tag, attrs)

Handle a start tag; override to act on it.

Parameters:

• tag (str) – the lowercased tag name.

• attrs (list[tuple[str, str | None]]) – the (name, value) attribute pairs, a valueless attribute’s value the empty string.

Return type:

None

handle_startendtag(tag, attrs)

Handle a self-closing tag; by default fire start then end, like the standard library.

Parameters:

• tag (str) – the lowercased tag name.

• attrs (list[tuple[str, str | None]]) – the (name, value) attribute pairs.

Return type:

None

handle_endtag(tag)

Handle an end tag; override to act on it.

Parameters:

tag (str) – the lowercased tag name.

Return type:

None

handle_data(data)

Handle a run of text, with character references resolved.

Parameters:

data (str) – the decoded text.

Return type:

None

handle_comment(data)

Handle a comment (also processing instructions and CDATA, which are comments here).

Parameters:

data (str) – the comment text, without the <!-- --> delimiters.

Return type:

None

handle_decl(decl)

Handle a <!DOCTYPE ...> declaration, with the leading <! and trailing > removed.

Parameters:

decl (str) – the declaration text, such as DOCTYPE html.

Return type:

None

handle_pi(data)

Handle a processing instruction; never called, since PIs are comments in the HTML spec.

Parameters:

data (str) – the instruction text.

Return type:

None

handle_entityref(name)

Handle a named reference; never called, since references are always resolved.

Parameters:

name (str) – the entity name, without the & and ;.

Return type:

None

handle_charref(name)

Handle a numeric reference; never called, since references are always resolved.

Parameters:

name (str) – the numeric reference body, without the &# and ;.

Return type:

None

unknown_decl(data)

Handle a CDATA section; never called, since it is a comment outside foreign content.

Parameters:

data (str) – the section text.

Return type:

None