Tokenize HTML

Run the WHATWG tokenizer over HTML: port an html.parser.HTMLParser subclass, or consume the token stream from turbohtml.tokenize() directly.

Migrate from html.parser

The quickest port keeps your subclass: turbohtml.migration.stdlib.HTMLParser is a drop-in base class with the same handle_* callbacks and feed/close methods, over the WHATWG-conformant tokenizer. Change the import and the base class and the handlers fire as before:

from turbohtml.migration.stdlib import HTMLParser


class LinkCollector(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links = []

    def handle_starttag(self, tag, attrs):
        if tag == "a":
            self.links += [v for n, v in attrs if n == "href" and v]


collector = LinkCollector()
collector.feed('<a href="/x">x</a> <a href="/y">y</a>')
collector.close()
print(collector.links)

['/x', '/y']

It differs from html.parser only where html.parser diverges from the WHATWG algorithm: references are always resolved (so handle_entityref/handle_charref never fire), and a processing instruction or CDATA section reaches handle_comment rather than handle_pi/unknown_decl, because the HTML spec treats both as comments.

If you would rather drop the subclass entirely, turbohtml also exposes the raw token stream. html.parser.HTMLParser is callback-driven: you subclass it and override a handler per event. turbohtml inverts that into a token stream you iterate, which removes the subclass, the mutable handler state, and the per-callback Python call overhead. A typical parser:

from html.parser import HTMLParser


class LinkCollector(HTMLParser):
    def __init__(self) -> None:
        super().__init__()
        self.links: list[str] = []

    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
        if tag == "a":
            self.links.extend(value for name, value in attrs if name == "href" and value)


collector = LinkCollector()
collector.feed(page)
collector.close()

becomes a loop:

import turbohtml

links = [
    href
    for token in turbohtml.tokenize(page)
    if token.type is turbohtml.TokenType.START_TAG and token.tag == "a" and (href := token.attr("href"))
]

The events map one to one:

  • handle_starttag(tag, attrs) → a token with type is TokenType.START_TAG; token.tag and token.attrs carry the same lowercased name and decoded (name, value) pairs, and token.attr(name) replaces scanning the list.

  • handle_endtag(tag) → TokenType.END_TAG.

  • handle_startendtag(tag, attrs) → a START_TAG token with self_closing true (turbohtml does not emit a separate event).

  • handle_data(data) → TokenType.TEXT; character references arrive decoded, like convert_charrefs=True, so there is no handle_entityref/handle_charref pair to implement.

  • handle_comment(data) → TokenType.COMMENT.

  • handle_decl(decl) → TokenType.DOCTYPE, split into name, public_id and system_id instead of one raw string.

  • self.getpos() → token.line and token.col, the same 1-based-line, 0-based-column convention.

  • feed()/close() → the same names on turbohtml.Tokenizer; each feed() returns the tokens that chunk completed instead of firing callbacks, and a with block replaces remembering close().

turbohtml differs from html.parser wherever html.parser diverges from the WHATWG algorithm browsers implement: turbohtml handles the raw-text content models (a <b> inside <script> stays text rather than a tag), recovers from malformed markup the way a browser would, and never emits handle_decl for CDATA sections (they only exist in foreign content). Code migrated from html.parser sees the same tokens a browser sees, the point of most migrations.

Extract the links of a page

Iterate the token stream and pull the href of every anchor start tag; turbohtml.Token.attr() returns the empty string for a valueless attribute and your fallback when the attribute is missing:

page = '<p><a href="/a">one</a> and <a href="/b" download>two</a></p>'
print([
    token.attr("href")
    for token in turbohtml.tokenize(page)
    if token.type is turbohtml.TokenType.START_TAG and token.tag == "a"
])

['/a', '/b']

Extract the visible text of HTML

Collect the text tokens while skipping the contents of elements the browser does not render, such as script and style. The tokenizer hands you script and style bodies as text tokens (that is what they are to the algorithm), so track the enclosing tag yourself:

from collections.abc import Iterator


def visible_text(page: str) -> Iterator[str]:
    hidden = 0
    for token in turbohtml.tokenize(page):
        if token.type is turbohtml.TokenType.START_TAG and token.tag in {"script", "style"}:
            hidden += 1
        elif token.type is turbohtml.TokenType.END_TAG and token.tag in {"script", "style"}:
            hidden -= 1
        elif token.type is turbohtml.TokenType.TEXT and not hidden:
            yield token.data


print("".join(visible_text("<style>p{}</style><p>Tom &amp; Jerry</p>")))

Tom & Jerry

Tokenize a document incrementally

When the input arrives in chunks, feed each chunk to a turbohtml.Tokenizer and consume the tokens it returns; text and unfinished tags stay buffered until they are complete, so the result is identical to tokenizing the whole string at once:

tokenizer = turbohtml.Tokenizer()
tokens = []
for chunk in ("<ul><li>on", "e<li>two</", "ul>"):
    tokens += tokenizer.feed(chunk)
tokens += tokenizer.close()
print([token.tag or token.data for token in tokens])

['ul', 'li', 'one', 'li', 'two', 'ul']

As a context manager the tokenizer signals end of input when the block exits, so forgetting close() cannot leave the final tokens stuck behind an unfinished construct; iterate the tokenizer itself to drain what remains:

with turbohtml.Tokenizer() as tokenizer:
    tokens = [token for chunk in ("<ul><li>on", "e") for token in tokenizer.feed(chunk)]
print([token.tag or token.data for token in tokenizer])

['one']

Call reset() to reuse the same tokenizer for an unrelated document.

Report source positions in diagnostics

Every token remembers where it began: turbohtml.Token.line is the 1-based source line and turbohtml.Token.col the 0-based column (the convention html.parser shares), so you can point at the offending markup:

page = "<h1>title</h1>\n<img src='a.png'>"
print([
    f"{token.tag} at {token.line}:{token.col}"
    for token in turbohtml.tokenize(page)
    if token.type is turbohtml.TokenType.START_TAG and token.tag == "img"
])

['img at 2:0']

See each character reference separately

By default a character reference is decoded into the surrounding text, so Tom &amp; Jerry is one text token reading Tom & Jerry. Pass resolve_references=False to turbohtml.tokenize() (or turbohtml.Tokenizer) to receive each reference in text as its own CHARACTER_REFERENCE token instead: data is the resolved value and source the verbatim &...; (so source[1] == "#" tells a numeric reference from a named one). A bare & that is not a reference stays text, and attribute values are always decoded:

import turbohtml
from turbohtml import TokenType

tokens = turbohtml.tokenize("5 &lt; 10 &amp; rising", resolve_references=False)
print([(token.data, token.source) for token in tokens if token.type is TokenType.CHARACTER_REFERENCE])

[('<', '&lt;'), ('&', '&amp;')]

Keep the verbatim source of a tag

The tokenizer normalizes tags: names lowercase, attribute order and quoting collapse. When you need the exact bytes a token came from - to rewrite markup in place, or to report it untouched - pass capture_source=True and read turbohtml.Token.source, the verbatim slice of the input. It is set for start tags, end tags, comments, and DOCTYPEs (text tokens leave it None):

tag = next(iter(turbohtml.tokenize("<IMG  SRC='a.png'>", capture_source=True)))
print(tag.tag, "from", repr(tag.source))

img from "<IMG  SRC='a.png'>"