####################### Tokenizing a document ####################### Go from a string of HTML to a stream of tokens you can inspect. Start with a small document and hand it to :func:`turbohtml.tokenize`, which returns an iterator of :class:`turbohtml.Token` objects: .. testcode:: import turbohtml for token in turbohtml.tokenize('

Tom & Jerry

'): print(token) .. testoutput:: Token(START_TAG, tag='p') Token(TEXT, data='Tom & Jerry') Token(END_TAG, tag='p') ``type`` identifies each token as a :class:`turbohtml.TokenType`. Start and end tags carry the lowercased tag name and the attributes, decoded: .. testcode:: start, text, end = turbohtml.tokenize('

Tom & Jerry

') print(start.type) print(start.tag) print(start.attrs) .. testoutput:: 1 p [('class', 'intro')] Text arrives with character references resolved (the ``&`` above came through as a plain ``&``). Content that the HTML specification treats as raw, such as a script body, arrives as one text token without further interpretation: .. testcode:: print([ token.data for token in turbohtml.tokenize("") if token.type is turbohtml.TokenType.TEXT ]) .. testoutput:: ['if (a < b) run()'] When the document arrives in pieces (from a network stream, for example), create a :class:`turbohtml.Tokenizer` and feed the pieces as they come. Each ``feed()`` returns the tokens that piece completed, and ``close()`` flushes whatever remains: .. testcode:: tokenizer = turbohtml.Tokenizer() print([token.tag for token in tokenizer.feed("
")]) print(list(tokenizer.close())) .. testoutput:: ['div'] ['span'] [] The incomplete ```` as raw text, and recovers from malformed markup the way a browser does. When the input is XML rather than HTML, reach for :func:`turbohtml.parse_xml` instead, which parses under XML 1.0 well-formedness -- names stay case-sensitive, ```` self-closes any element, and a CDATA section becomes its own node: .. testcode:: doc = turbohtml.parse_xml("Buy ]]>") root = doc.children[0] print(root.tag) print([child.tag for child in root if isinstance(child, turbohtml.Element)]) print(root.children[-1].data) .. testoutput:: Note ['Body'] A mismatched or unclosed tag is a well-formedness error there, not something to recover from -- it raises :exc:`~turbohtml.HTMLParseError` instead of building a repaired tree. Tokens are the raw stream; they know nothing about tree structure. When you want the events a browser's parser would fire -- with the implied ````, ````, and ```` filled in and stray table content foster-parented into place -- but you do not want to hold a tree, reach for :mod:`turbohtml.saxparse`. :func:`~turbohtml.saxparse.iter_events` parses a document and yields typed events in document order, materializing one at a time and keeping no tree: .. testcode:: from turbohtml.saxparse import iter_events for event in iter_events("stray`` no one wrote is there, and ``stray`` was moved out ahead of the ``
cell"): print(event) .. testoutput:: StartElement(tag='html', attrs=()) StartElement(tag='head', attrs=()) EndElement(tag='head') StartElement(tag='body', attrs=()) Characters(data='stray') StartElement(tag='table', attrs=()) StartElement(tag='tbody', attrs=()) StartElement(tag='tr', attrs=()) StartElement(tag='td', attrs=()) Characters(data='cell') EndElement(tag='td') EndElement(tag='tr') EndElement(tag='tbody') EndElement(tag='table') EndElement(tag='body') EndElement(tag='html') The ``
`` -- the tree the parser builds, delivered as events rather than nodes. If you prefer callbacks over a loop, subclass :class:`~turbohtml.saxparse.SaxHandler` and pass it to :func:`~turbohtml.saxparse.sax_parse`. When the thing you are building *is* a tree -- an index, a diff, or another library's nodes -- reach for :func:`turbohtml.treebuild.parse_into`. It runs the same tree builder and calls a builder object of yours for each node, handing you the parent so you assemble structure directly, with no tree of turbohtml's own. Here a builder skims the heading outline as the parse runs: .. testcode:: from turbohtml.treebuild import parse_into class Outline: def __init__(self): self.headings = [] def create_element(self, tag, namespace, attrs): if tag in {"h1", "h2", "h3"}: self.headings.append(tag) def create_document(self): ... def create_doctype(self, name, public_id, system_id): ... def create_text(self, data): ... def create_comment(self, data): ... def create_pi(self, data): ... def append(self, parent, child): ... outline = Outline() parse_into("

Title

x

Sub

", outline) print(outline.headings) .. testoutput:: ['h1', 'h2'] When you want to *change* the markup as it streams rather than just observe it, :func:`turbohtml.rewrite.rewrite` transforms a document in one pass without building a tree. You register a CSS selector and a handler; the handler edits each matching element in place, and everything you do not touch is copied through verbatim. Here every external link gains ``rel="noopener"`` while the local one is left exactly as written: .. testcode:: from turbohtml.rewrite import rewrite def external(link): if link.get("href", "").startswith("http"): link.set_attribute("rel", "noopener") html = '

See x and home.

' print(rewrite(html, elements=[("a[href]", external)])) .. testoutput::

See x and home.

The handler can also insert markup around an element, replace its inner content, unwrap it, or drop it, and separate handlers can rewrite text, comments, and the doctype. Because the pass never looks ahead, only selectors decidable from an element and its ancestors stream -- see :doc:`/how-to/rewriting` for the recipes and :doc:`/explanation/streaming` for why a sibling combinator or ``:nth-child`` cannot. The tree builder does more than fill in implied elements while it consumes the token stream. When it meets a ``