Tokenizing a document

Go from a string of HTML to a stream of tokens you can inspect.

Start with a small document and hand it to turbohtml.tokenize(), which returns an iterator of turbohtml.Token objects:

import turbohtml

for token in turbohtml.tokenize('<p class="intro">Tom &amp; Jerry</p>'):
    print(token)

Token(START_TAG, tag='p')
Token(TEXT, data='Tom & Jerry')
Token(END_TAG, tag='p')

type identifies each token as a turbohtml.TokenType. Start and end tags carry the lowercased tag name and the attributes, decoded:

start, text, end = turbohtml.tokenize('<p class="intro">Tom &amp; Jerry</p>')
print(start.type)
print(start.tag)
print(start.attrs)

1
p
[('class', 'intro')]

Text arrives with character references resolved (the &amp; above came through as a plain &). Content that the HTML specification treats as raw, such as a script body, arrives as one text token without further interpretation:

print([
    token.data
    for token in turbohtml.tokenize("<script>if (a < b) run()</script>")
    if token.type is turbohtml.TokenType.TEXT
])

['if (a < b) run()']

When the document arrives in pieces (from a network stream, for example), create a turbohtml.Tokenizer and feed the pieces as they come. Each feed() returns the tokens that piece completed, and close() flushes whatever remains:

tokenizer = turbohtml.Tokenizer()
print([token.tag for token in tokenizer.feed("<div><sp")])
print([token.tag for token in tokenizer.feed("an>")])
print(list(tokenizer.close()))

['div']
['span']
[]

The incomplete <sp stayed buffered until the rest of the tag arrived.

The tokenizer tracks the offset of every construct as it runs, and turbohtml.parse() can carry those offsets onto the tree it builds. Pass source_locations=True and each element’s source_location gives the span of its start tag, its end tag, and each attribute – the same information as parse5’s sourceCodeLocationInfo, here as SourceSpan records whose offsets slice the original source:

source = '<p class="intro">hi</p>'
element = turbohtml.parse(source, source_locations=True).find("p")
span = element.source_location.attrs["class"]
print(source[span.start_offset : span.end_offset])

class="intro"

The tokenizer follows the HTML rules throughout: it lowercases tag names, treats <script> as raw text, and recovers from malformed markup the way a browser does. When the input is XML rather than HTML, reach for turbohtml.parse_xml() instead, which parses under XML 1.0 well-formedness – names stay case-sensitive, <x/> self-closes any element, and a CDATA section becomes its own node:

doc = turbohtml.parse_xml("<Note><Body/>Buy <![CDATA[<milk>]]></Note>")
root = doc.children[0]
print(root.tag)
print([child.tag for child in root if isinstance(child, turbohtml.Element)])
print(root.children[-1].data)

Note
['Body']
<milk>

A mismatched or unclosed tag is a well-formedness error there, not something to recover from – it raises HTMLParseError instead of building a repaired tree.

Tokens are the raw stream; they know nothing about tree structure. When you want the events a browser’s parser would fire – with the implied <html>, <head>, and <body> filled in and stray table content foster-parented into place – but you do not want to hold a tree, reach for turbohtml.saxparse. iter_events() parses a document and yields typed events in document order, materializing one at a time and keeping no tree:

from turbohtml.saxparse import iter_events

for event in iter_events("<table>stray<tr><td>cell"):
    print(event)

StartElement(tag='html', attrs=())
StartElement(tag='head', attrs=())
EndElement(tag='head')
StartElement(tag='body', attrs=())
Characters(data='stray')
StartElement(tag='table', attrs=())
StartElement(tag='tbody', attrs=())
StartElement(tag='tr', attrs=())
StartElement(tag='td', attrs=())
Characters(data='cell')
EndElement(tag='td')
EndElement(tag='tr')
EndElement(tag='tbody')
EndElement(tag='table')
EndElement(tag='body')
EndElement(tag='html')

The <tbody> no one wrote is there, and stray was moved out ahead of the <table> – the tree the parser builds, delivered as events rather than nodes. If you prefer callbacks over a loop, subclass SaxHandler and pass it to sax_parse().

When the thing you are building is a tree – an index, a diff, or another library’s nodes – reach for turbohtml.treebuild.parse_into(). It runs the same tree builder and calls a builder object of yours for each node, handing you the parent so you assemble structure directly, with no tree of turbohtml’s own. Here a builder skims the heading outline as the parse runs:

from turbohtml.treebuild import parse_into


class Outline:
    def __init__(self):
        self.headings = []

    def create_element(self, tag, namespace, attrs):
        if tag in {"h1", "h2", "h3"}:
            self.headings.append(tag)

    def create_document(self): ...
    def create_doctype(self, name, public_id, system_id): ...
    def create_text(self, data): ...
    def create_comment(self, data): ...
    def create_pi(self, data): ...
    def append(self, parent, child): ...


outline = Outline()
parse_into("<h1>Title</h1><p>x</p><h2>Sub</h2>", outline)
print(outline.headings)

['h1', 'h2']

When you want to change the markup as it streams rather than just observe it, turbohtml.rewrite.rewrite() transforms a document in one pass without building a tree. You register a CSS selector and a handler; the handler edits each matching element in place, and everything you do not touch is copied through verbatim. Here every external link gains rel="noopener" while the local one is left exactly as written:

from turbohtml.rewrite import rewrite


def external(link):
    if link.get("href", "").startswith("http"):
        link.set_attribute("rel", "noopener")


html = '<p>See <a href="https://x.test">x</a> and <a href="/local">home</a>.</p>'
print(rewrite(html, elements=[("a[href]", external)]))

<p>See <a href="https://x.test" rel="noopener">x</a> and <a href="/local">home</a>.</p>

The handler can also insert markup around an element, replace its inner content, unwrap it, or drop it, and separate handlers can rewrite text, comments, and the doctype. Because the pass never looks ahead, only selectors decidable from an element and its ancestors stream – see Rewrite HTML without building a tree for the recipes and The streaming rewrite model for why a sibling combinator or :nth-child cannot.

The tree builder does more than fill in implied elements while it consumes the token stream. When it meets a <template> carrying a shadowrootmode, it attaches a declarative shadow root to the template’s parent and parses the template’s content into that shadow tree instead of a template content fragment – the same markup a browser turns into a shadow root. The template element itself never lands in the light tree, so the parent serializes without it:

from turbohtml import parse

card = parse("<div id=card><template shadowrootmode=open><slot></slot></template><p>Body</p></div>").find(id="card")
print(card.shadow_root.mode)
print(card.html)

open
<div id="card"><p>Body</p></div>

Declarative shadow roots are honored for whole-document parse() by default; pass allow_declarative_shadow_roots=False to keep such templates as ordinary elements. See Use the Shadow DOM for working with the shadow tree the parser built.

That is the whole tokenizer API. If you are porting an existing html.parser.HTMLParser subclass, turbohtml.migration.stdlib.HTMLParser keeps the same handle_* callbacks over this tokenizer, so the migration is changing the base class. Head to the How-to guides guides for task-focused recipes or the Reference for the exact signatures.