.. _migration-lxml: ########### From lxml ########### .. package-meta:: lxml lxml/lxml `lxml `_ is the libxml2/libxslt binding that most Python HTML and XML processing has been built on. ``lxml.html`` parses documents into ElementTree-style elements with ``.text``/``.tail`` strings, and the wider stack adds XPath, XSLT, RelaxNG/DTD/XML-Schema validation, and C14N. It is the default reach for scraping, feed parsing, and XML pipelines because it wraps a fast, battle-tested C library and exposes the full ElementTree API. turbohtml covers the HTML side of that ground with a native C core of its own. :func:`turbohtml.parse` builds the WHATWG document tree libxml2's HTML parser does not, returns a fully type annotated :class:`~turbohtml.Document`, and folds XPath 1.0 (with EXSLT), CSS selection, and the ``find``/``find_all`` grammar into one node API instead of separate ``findall``/``xpath``/``cssselect`` entry points. It does not attempt XSLT, schema validation, or generic XML; it targets browser-accurate HTML parsing and the query/edit/serialize surface around it. ******************* turbohtml vs lxml ******************* .. list-table:: :header-rows: 1 :widths: 20 40 40 - - Dimension - turbohtml - lxml - - Scope - WHATWG HTML5 parse, serialize, edit, CSS, XPath 1.0 + EXSLT, link helpers - Generic XML and HTML via libxml2, plus XSLT, schema validation, C14N - - Feature breadth - Browser-accurate HTML tree, one node API for XPath/CSS/find, streaming parse, builder - Full ElementTree API, XPath 1.0, XSLT 1.0, DTD/RelaxNG/XML-Schema, iterparse - - Performance - Parses two to four times faster than lxml; stays ahead across the operational surface - Mature libxml2 C core; streaming evaluation narrows on multi-megabyte inputs - - Typing - Fully type annotated, ships stubs - Partial; relies on third-party stub packages - - Dependencies - Self-contained native C extension - Bundles or links libxml2 and libxslt - - Maintenance - Newer, WHATWG-spec-driven - Long-established, widely deployed, actively maintained Feature overlap =============== These port one-to-one from ``lxml.html``/``lxml.etree`` to turbohtml: - Parsing a document (``lxml.html.document_fromstring``) and a fragment (``lxml.html.fromstring``). - Element identity and attributes: ``el.tag``, ``el.get``/``el.set``/``el.attrib``, the ``el.classes`` set operations. - Tree navigation: ``getparent``, ``getnext``, ``getprevious``, ``iterdescendants``, ``iterancestors``, ``list(el)``. - Queries: ``findall`` and ``xpath`` (XPath 1.0), ``cssselect`` (CSS), precompiled ``etree.XPath`` objects, node-set ``$variable`` bindings, ``namespaces=`` prefix maps, and custom XPath callables. - The EXSLT ``re:``, ``set:``, ``str:``, ``math:``, and ``date:`` function namespaces. - Locator generation (``getroottree().getpath``), link iteration and rewriting (``iterlinks``, ``make_links_absolute``, ``rewrite_links``), source positions (``sourceline``), tree edits (``drop_tag``, ``drop_tree``), the ``lxml.builder.E`` builder, and serialization (``lxml.html.tostring``). What turbohtml adds =================== - A WHATWG-conformant parse: malformed input lands in the same tree a browser builds, where libxml2's HTML parser does not follow the HTML5 tree-construction algorithm. - One node API. XPath, CSS, and the ``find``/``find_all`` grammar are methods on every node rather than three separate extension entry points, and a callable or ``extensions=`` mapping that returns an :class:`~turbohtml.Element` is marshaled straight back into the evaluator's node-set. - Built-in EXSLT. The ``re:``, ``set:``, ``str:``, ``math:``, and ``date:`` namespaces dispatch in the compiled-C XPath engine with no per-call registration; lxml has to register ``libexslt`` and re-resolve the namespace map on each evaluation. - :meth:`~turbohtml.Element.css_path`, a unique CSS-selector locator, which lxml has no equivalent for. - Full type annotations and shipped stubs across the whole surface. What lxml has that turbohtml does not ===================================== The wider libxml2 toolchain is a deliberate clean-break scope cut: - XSLT (``lxml.etree.XSLT``): no equivalent. - Schema validation (DTD, RelaxNG, XML-Schema, Schematron): no equivalent. - C14N canonicalization: no equivalent. - Generic XML parsing and namespaced XML documents: turbohtml targets HTML; use lxml for arbitrary XML. - XPath is at parity, not a gap. Both are XPath 1.0, and both run EXSLT. The only pieces out of scope are the node-synthesizing ``str:tokenize``/``str:split`` and the implicit current-date ``date:`` forms. Performance =========== turbohtml parses two to four times faster than lxml while matching a browser on malformed input, and stays ahead across the operational surface: fragment parsing, CSS selection, text and tree walks, the link helpers, XPath, and the node-path generators. .. bench-table:: :file: bench/lxml.json The :doc:`/development/performance` page benchmarks the full serializer, builder, editor, CSS, XPath 1.0, and EXSLT surface against lxml directly, and sweeps the node-path generators across every page size. Compiling a hot expression once with :class:`~turbohtml.XPath` (the parse happens at construction, so the call site only supplies the context node and any ``$name`` variables) stays ahead of lxml per evaluation, as the precompiled ``//a[@href]`` row shows. On the EXSLT cases, a ``re:test`` predicate runs nearly twenty times ahead of lxml even though ``re:`` dispatches to Python's :mod:`re` where lxml uses C ``libexslt``, because it skips the per-call namespace resolution; lxml's streaming evaluation narrows the node-set reductions on the multi-megabyte inputs. **************** How to migrate **************** The two parse entry points swap directly: :func:`turbohtml.parse` replaces ``lxml.html.document_fromstring`` and :func:`turbohtml.parse_fragment` replaces ``lxml.html.fromstring``. The biggest change is the tree shape. lxml stores text as an element's ``.text`` and ``.tail`` strings; turbohtml models it as real child :class:`~turbohtml.Text` nodes, so you iterate :attr:`~turbohtml.Node.children` instead of reading two string fields. .. list-table:: :header-rows: 1 :widths: 50 50 - - `lxml `__ - turbohtml - - ``el.tag`` - :attr:`~turbohtml.Element.tag` (same) - - ``el.get("x")``, ``el.attrib``, ``el.set("x", "v")`` - :attr:`~turbohtml.Element.attrs` (``attrs.get("x")``, ``attrs["x"] = "v"``) - - ``el.classes.add("x")``, ``el.classes.discard("x")``, ``el.classes.toggle("x")``, ``"x" in el.classes`` - :meth:`el.add_class("x") `, :meth:`el.remove_class("x") `, :meth:`el.toggle_class("x") `, :meth:`el.has_class("x") ` - - ``el.text``, ``el.tail`` - child :class:`~turbohtml.Text` nodes; iterate :attr:`~turbohtml.Node.children` - - ``el.text_content()`` - :attr:`~turbohtml.Node.text` - - ``el.getparent()``, ``el.getnext()``, ``el.getprevious()`` - :attr:`~turbohtml.Node.parent`, :attr:`~turbohtml.Node.next_sibling`, :attr:`~turbohtml.Node.previous_sibling` - - ``list(el)``, ``el.iterdescendants()``, ``el.iterancestors()`` - :attr:`~turbohtml.Node.children`, :attr:`~turbohtml.Node.descendants`, :attr:`~turbohtml.Node.ancestors` - - ``el.findall(".//a")``, ``el.xpath("//a[@href]")`` - :meth:`~turbohtml.Node.find_all`, :meth:`~turbohtml.Node.xpath` - - ``etree.XPath("//a[@href=$u]")(el, u=v)`` - :class:`~turbohtml.XPath` (``XPath("//a[@href=$u]")(el, u=v)``) - - ``el.xpath("$rows/td", rows=el.xpath("//tr"))`` - :meth:`el.xpath("$rows/td", rows=el.xpath("//tr")) ` (a ``$name`` variable binds a scalar, an :class:`~turbohtml.Element`, or an iterable of elements; :meth:`~turbohtml.Node.xpath_one` and :meth:`~turbohtml.Node.xpath_iter` take the same bindings) - - ``el.xpath("//svg:rect", namespaces={"svg": SVG})`` - :meth:`~turbohtml.Node.xpath` with the same ``namespaces={"svg": SVG}`` (the prefix binds at evaluation time) - - ``el.cssselect("div a")`` - :meth:`~turbohtml.Node.select` - - ``etree.FunctionNamespace(None)["f"] = fn``; ``el.xpath("f(//a)")`` - :meth:`el.xpath("f(//a)", extensions={(None, "f"): fn}) ` (the function may return a scalar, an :class:`~turbohtml.Element`, or an iterable of elements) - - ``el.getroottree().getpath(el)`` - :meth:`el.xpath_path() ` (or :meth:`el.css_path() ` for a CSS selector) - - ``lxml.html.Element("div")``, ``etree.SubElement(p, "div")`` - :class:`~turbohtml.Element`, :meth:`p.append(Element("div")) ` - - ``lxml.builder.E.ul(E.li("a"), E.li("b"))`` - :data:`turbohtml.build.E` (``E.(attrs, *children)`` with a leading attribute mapping) - - ``el.drop_tag()``, ``el.drop_tree()`` - :meth:`~turbohtml.Node.unwrap`, :meth:`~turbohtml.Node.decompose` - - ``el.sourceline`` - :attr:`~turbohtml.Node.source_line` (1-based, like lxml; plus :attr:`~turbohtml.Node.source_col`) - - ``el.iterlinks()`` - :meth:`~turbohtml.Node.links` - - ``el.make_links_absolute(base)``, ``el.rewrite_links(fn)`` - :meth:`~turbohtml.Node.resolve_links`, :meth:`~turbohtml.Node.rewrite_links` - - ``etree.iterparse(...)`` - :class:`turbohtml.IncrementalParser` (``feed`` chunks, ``close`` for the :class:`~turbohtml.Document`) - - ``lxml.html.tostring(el)`` - :attr:`~turbohtml.Node.html` A query-and-select flow ports directly: .. testcode:: doc = parse('') print(doc.find_all("a", attrs={"href": True})) print(doc.select_one("div a").attrs["href"]) .. testoutput:: [Element('a')] /x Precompile a hot XPath the same way you would reach for ``lxml.etree.XPath`` over a bare ``el.xpath``. turbohtml's compiled program is tree-independent, so a single object evaluates against many documents: .. testcode:: from turbohtml import XPath links = XPath("//a[@href=$u]") doc = parse('') print([link.attrs["href"] for link in links(doc, u="/x")]) .. testoutput:: ['/x'] The builder reads like ``lxml.builder.E`` but hands back a real :class:`~turbohtml.Element`, so the query, edit, and serialize surface stays available on what you build: .. testcode:: from turbohtml.build import E print(E.ul(E.li({"class": "item"}, "one"), E.li({"class": "item"}, "two")).serialize()) .. testoutput::
  • one
  • two
********************** Gotchas and pitfalls ********************** - No ``text``/``tail``. A node's children are its text runs and elements interleaved; read :attr:`~turbohtml.Node.text` for the concatenation. - Different tree on malformed input. lxml parses with libxml2, which is not WHATWG-conformant, so broken markup lands in a different tree than the one turbohtml (and a browser) builds. Do not expect byte-identical trees when porting scrapers that leaned on libxml2's recovery quirks. - Custom XPath functions bind per call, not globally. lxml registers callables through ``etree.FunctionNamespace``; turbohtml passes them through the ``extensions=`` mapping of :meth:`~turbohtml.Node.xpath`, bound once against the compiled expression rather than a process-wide table. - Streaming differs. For a document that arrives in pieces, ``etree.iterparse`` is replaced by :class:`turbohtml.IncrementalParser`: feed ``str`` or ``bytes`` chunks with ``feed`` and call ``close`` for the finished :class:`~turbohtml.Document`. It never holds the whole source at once, but it does not expose lxml's event-driven element callbacks; you walk the completed tree. - EXSLT is built in but not exhaustive. The node-synthesizing ``str:tokenize``/``str:split`` and the implicit current-date ``date:`` forms stay out of scope; every other ``re:``/``set:``/``str:``/``math:``/``date:`` form ports straight through with no registration.