Query a tree with XPath

Evaluate XPath 1.0 against a node with xpath(), xpath_one(), and xpath_iter(): the axes and node tests, $name variables, namespaces, the EXSLT function sets, custom extensions, and the compiled turbohtml.XPath object.

xpath() evaluates an XPath 1.0 expression relative to a node and returns a list for a node-set (elements as nodes, attribute and text() values as str, in document order), or the matching float / str / bool for a scalar expression like count(...) or string(...). xpath_one() returns the first result or None, and xpath_iter() returns an iterator. The engine supports the structural axes, the name / * / node() / text() / comment() / processing-instruction() node tests, predicates, the boolean, relational, and arithmetic operators, unions, and the complete XPath 1.0 core function library:

import turbohtml

doc = turbohtml.parse("<table><tr><td>1</td><td>2</td></tr><tr><td>3</td><td>4</td></tr></table>")
print([td.text for td in doc.xpath("//td")])
print(doc.xpath("//tr[2]/td[1]/text()"))
print(doc.xpath("count(//td)"))
print(doc.xpath_one("//td[. = '3']").text)

['1', '2', '3', '4']
['3']
4.0
3

An absolute path starts at the document root and a leading // rescans the whole document, so write .// for descendants of the context node. Migrating from lxml, parsel, or pyquery keeps your existing expressions.

Two functions read the HTML document the way HTML means it, where lxml’s legacy HTML parser returns nothing: lang() honors the HTML lang attribute (lxml only consults xml:lang), and namespace-uri() reports the real SVG and MathML namespace for foreign content (lxml leaves it empty). HTML elements report no namespace in both, so an unprefixed name test keeps matching them.

To select foreign content by namespace, bind each prefix to a URI through namespaces (the same argument lxml takes). A prefixed name test then matches an element whose namespace equals the bound URI and whose local name equals the suffix, so //svg:rect finds the SVG rectangle while the unprefixed //rect keeps matching by name alone:

import turbohtml

doc = turbohtml.parse("<svg><rect/><circle/></svg>")
rects = doc.xpath("//svg:rect", namespaces={"svg": "http://www.w3.org/2000/svg"})
print(len(rects))

1

Pass $name variables as keyword arguments instead of formatting values into the expression string, so a value with quotes or special characters cannot break the query:

import turbohtml

doc = turbohtml.parse("<a href='/in'>in</a><a href='/out'>out</a>")
print([a.text for a in doc.xpath("//a[@href=$href]", href="/out")])

['out']

A variable can also hold a node-set: bind an Element or any iterable of elements (a prior xpath() result) and feed it back into a later expression instead of re-walking the tree. The node-set joins path steps, count(), unions, and predicates like any other node-set:

import turbohtml

doc = turbohtml.parse("<table><tr><td>a</td><td>b</td></tr><tr><td>c</td></tr></table>")
rows = doc.xpath("//tr")
print([td.text for td in doc.xpath("$rows/td", rows=rows)])
print(doc.xpath("count($rows)", rows=rows))

['a', 'b', 'c']
2.0

The EXSLT re:test and re:replace functions parsel and scrapy rely on work without registering a namespace; the re: prefix dispatches to Python’s re:

import turbohtml

doc = turbohtml.parse("<a href='/p/12'>a</a><a href='/q'>b</a>")
print([a.attrs["href"] for a in doc.xpath(r"//a[re:test(@href, '\d')]")])

['/p/12']

Register your own functions under extensions={(namespace, name): callable} (use None for the namespace to call the function unprefixed). The callable receives a context whose context_node is the current element, then the evaluated arguments: a node-set arrives as a list of Element, and a string, number, or boolean as the matching Python scalar. Returning a str, number, or bool produces a scalar; returning an element or an iterable of elements produces a node-set that feeds later path steps and predicates, exactly as in lxml:

import turbohtml
from types import SimpleNamespace
from turbohtml import Element


def first_two(context: SimpleNamespace, nodes: list[Element]) -> list[Element]:
    return nodes[:2]


doc = turbohtml.parse("<ul><li>a</li><li>b</li><li>c</li></ul>")
result = doc.xpath("first_two(//li)/text()", extensions={(None, "first_two"): first_two})
print(result)

['a', 'b']

Every element in a returned node-set must belong to the document being queried; returning one from another parse raises ValueError.

When one expression runs over many nodes or documents – a scraper looping rows, or one query across a corpus – compile it once with turbohtml.XPath instead of re-parsing it on every xpath() call. Bind smart_strings and extensions at construction, then call the object with a context node and any $name variables; it returns exactly what the matching xpath() call would. The compiled object is immutable and re-entrant, so one instance can be shared across threads.

import turbohtml

doc = turbohtml.parse("<table><tr><td class='num'>1</td><td>x</td></tr><tr><td class='num'>2</td></tr></table>")
cells = turbohtml.XPath(".//td[@class=$cls]")
for row in doc.xpath("//tr"):
    print([td.text for td in cells(row, cls="num")])

['1']
['2']

The same built-in dispatch covers the EXSLT set:, str:, math:, and date: namespaces, so the node-set, string, numeric, and date helpers lxml gets from libexslt work without registering anything. set: operates on node-sets (difference, intersection, distinct, has-same-node, leading, trailing), str: builds strings (replace literal-not-regex, concat, padding, align), math: reduces node-sets and numbers (min, max, highest, lowest, abs, power), and date: reads fields out of an ISO YYYY-MM-DD string (year, month-in-year, day-in-month, day-in-week, leap-year):

import turbohtml

doc = turbohtml.parse("<ul><li>3</li><li>1</li><li>1</li></ul>")
print([li.text for li in doc.xpath("set:distinct(//li)")])
print(doc.xpath("math:max(//li)"))
print(doc.xpath("str:padding(3, '-')"))
print(doc.xpath("date:year('2024-06-22')"))

['3', '1']
3.0
---
2024.0

str:tokenize and str:split are not built in: they would have to synthesize token nodes, and the engine’s node-sets only reference nodes that already exist in the tree. Likewise date: reads an explicit date string rather than the implicit current date-time, so a query stays deterministic. Register either through extensions= if you need it.