Parsing a document into a tree

A token stream is flat. To see which element contains which, you need the structure: a tree. Go from a string of HTML to a navigable tree of nodes.

Important

The one rule worth learning first: turbohtml models text as real child nodes (the WHATWG DOM shape), not lxml’s text/tail or BeautifulSoup’s .string. So node[i] indexes a node’s children, and attributes are reached through node.attrs, never node["attr"].

Hand a whole document to turbohtml.parse(). It applies the full WHATWG tree-construction algorithm (the same one browsers run, including the error recovery that inserts the missing html, head and body) and returns a turbohtml.Document:

import turbohtml

doc = turbohtml.parse("<h1>Hello</h1><p>Tom &amp; <a href='/x'>Jerry</a></p>")
print(doc.root)

Element('html')

The recovery is not silent: each WHATWG parse error turbohtml recovered from is on errors, a list of ParseError with the spec code and source position. A clean document leaves it empty; malformed input fills it (and parse(..., strict=True) raises HTMLParseError on the first one):

print(doc.errors)
print(turbohtml.parse("<a b b>").errors[0].code)

[]
duplicate-attribute

By default turbohtml parses <noscript> content as markup, so you can walk into it. A scripting browser treats that content as raw text instead. Pass scripting=True to build that tree, where the inner tags become one text run:

noscript = turbohtml.parse("<noscript><a href='/no-js'>plain</a></noscript>", scripting=True).find("noscript")
print(noscript.text)

<a href='/no-js'>plain</a>

find() returns the first descendant matching a tag (and any attributes you pass), or None:

print(doc.find("a"))
print(doc.find("a").attrs)

Element('a')
{'href': '/x'}

Every node exposes its text and its markup. text is the concatenated character data of the subtree, with references decoded; html re-serializes the subtree:

paragraph = doc.find("p")
print(paragraph.text)
print(paragraph.html)

Tom & Jerry
<p>Tom &amp; <a href="/x">Jerry</a></p>

turbohtml models text as real child nodes (the WHATWG DOM shape), so a paragraph’s children are its text runs and its elements interleaved, in order. A node is a sequence of its children: iterate it, take its length, index into it:

print(list(paragraph))
print(len(paragraph))
print(paragraph[1])

[Text('Tom & '), Element('a')]
2
Element('a')

From any node you can walk outward as well as inward: parent, next_sibling, and the lazy ancestors and descendants iterators:

link = doc.find("a")
print(link.parent)
print([node.tag for node in link.ancestors if isinstance(node, turbohtml.Element)])

Element('p')
['p', 'body', 'html']

For richer queries, select() takes a CSS selector and returns every matching descendant in document order. The negation pseudo-class :not() keeps the elements that match none of its arguments; here, the descendants of body that are not links:

print([node.tag for node in doc.select("body :not(a)")])

['h1', 'p']

Selectors also reach the form and UI pseudo-classes the markup determines, such as :checked for a checked control:

form = turbohtml.parse("<input type=checkbox checked><input type=checkbox>")
print(len(form.select(":checked")))

1

:is() and :where() are forgiving, so an arm they cannot parse is dropped and the rest still select; a typo in one alternative does not break the query:

print([node.tag for node in doc.select(":is(h1, :oops)")])

['h1']

Structural pseudo-classes count positions, and :nth-child(An+B of S) counts only the siblings matching S; here the first checked box, ignoring the unchecked ones in between:

boxes = turbohtml.parse("<p><input checked><input><input checked></p>")
print([e.attrs.get("checked") for e in boxes.select("input:nth-child(1 of [checked])")])

['']

If you are coming from pyquery’s jQuery-style chaining, turbohtml.query.Query wraps these primitives in a fluent, chainable surface where each call returns a new wrapper.

Because the node types are a sealed hierarchy, structural pattern matching works: each subtype unpacks its defining field:

for node in paragraph:
    match node:
        case turbohtml.Element(tag):
            print("element", tag)
        case turbohtml.Text(data):
            print("text", repr(data))

text 'Tom & '
element a

Query with XPath

When you are porting a scraper written against lxml or elementpath, xpath() runs the expression as-is. Beyond the XPath 1.0 core it also answers the string subset of XPath 2.0, so upper-case folds a result’s case and ends-with filters inside a predicate:

print(doc.xpath("upper-case(//a)"))
print([a.attrs["href"] for a in doc.xpath("//a[ends-with(@href, '/x')]")])

JERRY
['/x']

Chain queries fluently

When you would rather chain than write a comprehension, wrap the same tree in turbohtml.query.Query. Each call narrows the selection and returns a new wrapper, so you reach the link’s text and target without an intermediate variable:

from turbohtml.query import Query

anchors = Query(doc).find("a")
print(anchors.text())
print(anchors.attr("href"))

Jerry
/x

Walk with a filtered cursor

The descendants iterator visits every node once, front to back. When you want a cursor you can steer – step to a child, a sibling, or back to the parent, and point at a filtered subset – reach for the DOM turbohtml.TreeWalker. Its what_to_show bitmask picks node types (here SHOW_ELEMENT, so text and comments are stepped over), and each move returns the node it landed on:

from turbohtml import NodeFilter, TreeWalker

walker = TreeWalker(paragraph, NodeFilter.SHOW_ELEMENT)
print(walker.first_child())
print(walker.next_node())

Element('a')
None

A filter callback refines the walk further. It returns one of three verdicts, and the difference between two of them is what makes it a tree walker: FILTER_REJECT drops a node and its subtree, while FILTER_SKIP drops only the node, so the walk still descends into its children:

tree = turbohtml.parse("<section><aside><b>x</b></aside><i>y</i></section>").find("section")


def hide_aside(verdict):
    return lambda node: verdict if node.tag == "aside" else NodeFilter.FILTER_ACCEPT


skip = TreeWalker(tree, NodeFilter.SHOW_ELEMENT, hide_aside(NodeFilter.FILTER_SKIP))
reject = TreeWalker(tree, NodeFilter.SHOW_ELEMENT, hide_aside(NodeFilter.FILTER_REJECT))
print([node.tag for node in iter(skip.next_node, None)])
print([node.tag for node in iter(reject.next_node, None)])

['b', 'i']
['i']

For a flat, front-to-back view of the same filtered nodes, turbohtml.NodeIterator iterates directly; it has no subtree to prune, so it treats reject and skip alike. See Walk a tree with a cursor and filter for the full cursor and Traversal objects for why the reject/skip split matters.

Copy a slice with a Range

To grab a run of siblings rather than a single node, mark it with a turbohtml.Range. A range holds two boundary points – each a (container, offset) pair – and clone_contents() returns a fragment copying everything between them, leaving the tree untouched. Span the paragraph’s second child (the link) and copy it:

from turbohtml import Range

para = doc.find("p")
span = Range(para, 1)
span.set_end(para, 2)
print(span.clone_contents().html)

<a href="/x">Jerry</a>

Compose a slot with a shadow root

An element can host a shadow tree: a private subtree, kept off to the side, whose <slot> elements pull in the host’s own children. attach_shadow() returns the turbohtml.ShadowRoot; fill it with slots, and each light-DOM child is assigned to the slot whose name matches its slot attribute, with the unnamed default slot taking the rest. flattened_children then shows the composed result – the shadow tree with every slot replaced by what it received:

from turbohtml import Element, Text

card = Element("article")
card.append(Element("h2", {"slot": "title"}, [Text("Shadow DOM")]))
card.append(Element("p", None, [Text("Composed at render time.")]))

root = card.attach_shadow("open")
root.set_inner_html('<header><slot name="title"></slot></header><main><slot></slot></main>')

title = root.select_one('slot[name="title"]')
print([node.tag for node in title.assigned_nodes()])
print([node.tag for node in card.flattened_children])

['h2']
['header', 'main']

The shadow tree stays out of the host’s own children and serialization; it is reachable only through shadow_root (or the reference attach_shadow() returned). See Use the Shadow DOM for the slot recipes and Shadow DOM for the flattened-tree model.

Continue to Building and editing a tree to build and change trees of your own.