From BeautifulSoup¶

BeautifulSoup is the long-standing convenience layer over a choice of HTML parsers (html.parser, lxml, or html5lib): you pick a backend, then navigate and search the resulting soup with a large, alias-rich API. It also parses XML through the lxml-xml backend, decodes unknown byte streams with UnicodeDammit, and matches CSS selectors through its soupsieve dependency. Its reach and forgiving API made it the default scraping tool for a generation of Python code, from one-off screen scrapes to production pipelines.

turbohtml covers the HTML side of that ground with one engine. turbohtml.parse() runs the WHATWG algorithm in C, returns a fully type annotated Document, and exposes a single find/find_all/select grammar plus XPath. It shares the most surface with turbohtml of any library here, so this is the deepest migration section.

turbohtml vs BeautifulSoup¶

Dimension	turbohtml	BeautifulSoup
Scope	WHATWG HTML: parse, query, mutate, and serialize in one C engine	HTML and XML navigation layer over a pluggable third-party parser
Feature breadth	`find` grammar with `Axis`, CSS selectors, XPath, source positions, linkify, WHATWG serialization formatters	`find` grammar, CSS via `soupsieve`, XML mode, `UnicodeDammit` encoding guessing, output-formatter registry
Performance	C core; one to three orders of magnitude faster than `bs4` over `html.parser` across parse, query, and serialize	Pure-Python navigation; speed tracks the chosen backend and is slowest on `html.parser`
Typing	Ships `py.typed`; every public method annotated	No inline types; relies on the third-party `types-beautifulsoup4` stubs
Dependencies	Zero runtime dependencies (self-contained C extension)	Requires `soupsieve`; `lxml` or `html5lib` optional for those backends
Maintenance	Actively developed, single-engine	Mature, widely deployed, long release history

Feature overlap¶

These port one-to-one; the calls differ only in name (see the mapping table under How to migrate):

Parsing markup into a navigable tree.
find / find_all by tag, attribute, class_, and text.
CSS selectors via select / select_one.
Directional navigation (parents, siblings, next/previous elements) — turbohtml folds these into find with an Axis.
Tree mutation: decompose, extract, unwrap, wrap, insert_before, insert_after, replace_with.
Text access: get_text maps to text, strings, and stripped_strings.
Pretty-printing via serialize() with Html(layout=Indent(2)) for prettify().
Source positions: sourceline / sourcepos map to source_line / source_col.

What turbohtml adds¶

XPath. xpath() and xpath_iter() evaluate expressions with namespaces, variables, and extension functions. BeautifulSoup has no XPath support at all.
A C engine. Parsing, querying, text collection, and serialization all run in C, so migrating off html.parser is a large speedup with no backend to install.
WHATWG-conformant serialization. Formatter selection controls entities, and output matches the spec by default; Formatter.NAMED_ENTITIES reproduces bs4’s html formatter when you need it.
Post-parse pruning. prune() trims a fully parsed tree to a CSS selector in one C pass.
Zero runtime dependencies and full typing. No soupsieve install, and every public API is annotated.

What BeautifulSoup has that turbohtml does not¶

XML parsing. BeautifulSoup(markup, "lxml-xml") parses arbitrary XML. turbohtml runs the WHATWG HTML algorithm only; there is no XML tree-builder. No equivalent — keep lxml for XML documents.
Pluggable parser backends. bs4 lets you swap html.parser, lxml, and html5lib. turbohtml always runs its own WHATWG parser; this is a deliberate clean-break omission, not a gap you work around.
Statistical encoding detection. UnicodeDammit (and chardet / charset-normalizer) guess an encoding from byte frequency when there is no BOM or <meta charset>. turbohtml sniffs only what the WHATWG algorithm reads, then falls back to windows-1252. Workaround: detect with charset-normalizer first and hand turbohtml the decoded str (or bytes with an explicit encoding=).
A named output-formatter registry. bs4 registers custom formatters by name. Workaround: pass a Formatter per serialize() call.

Performance¶

turbohtml parses, queries, and serializes one to three orders of magnitude faster than BeautifulSoup over html.parser — including text filtering (find(text=...) against find_all(string=...)), walking the tree (descendants against soup.descendants), and reading text (text against soup.get_text()):

operation	turbohtml	BeautifulSoup
build a list (constructors) — 100 rows	55.7 µs	1.03 ms (18.6x)
build a list (constructors) — 1k rows	566 µs	8.63 ms (15.3x)
build a list (constructors) — 10k rows	6.41 ms	95.4 ms (14.9x)
construct N elements (no serialize) — 100 rows	43.4 µs	265 µs (6.2x)
construct N elements (no serialize) — 1k rows	463 µs	2.59 ms (5.6x)
construct N elements (no serialize) — 10k rows	4.66 ms	25.4 ms (5.5x)
emit a built tree — 100 rows	4.19 µs	394 µs (94.0x)
emit a built tree — 1k rows	42.4 µs	4.01 ms (94.8x)
emit a built tree — 10k rows	420 µs	41.7 ms (99.4x)
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	60.8 µs (50.5x)
parse to a tree — wpt small (4 kB)	9.57 µs	435 µs (45.5x)
parse to a tree — wpt medium (9.6 kB)	24.2 µs	853 µs (35.3x)
parse to a tree — wpt large (92 kB)	207 µs	15.1 ms (73.0x)
parse to a tree — wpt CJK (124 kB)	407 µs	21.9 ms (53.9x)
parse to a tree — whatwg spec (235 kB)	405 µs	25.5 ms (62.9x)
find every anchor — daring fireball (10 kB)	375 ns	13.5 µs (36.2x)
find every anchor — ars technica (56 kB)	828 ns	50.5 µs (61.1x)
find every anchor — mozilla blog (95 kB)	1.18 µs	107 µs (90.9x)
find every anchor — whatwg spec (235 kB)	1.32 µs	393 µs (299x)
select div a[href] — daring fireball (10 kB)	639 ns	168 µs (264x)
select div a[href] — ars technica (56 kB)	1.46 µs	566 µs (388x)
select div a[href] — mozilla blog (95 kB)	2.07 µs	1.07 ms (519x)
select div a[href] — whatwg spec (235 kB)	1.74 µs	2.74 ms (1575x)
select div:has(a) — daring fireball (10 kB)	256 ns	112 µs (438x)
select div:has(a) — ars technica (56 kB)	1.28 µs	486 µs (381x)
select div:has(a) — mozilla blog (95 kB)	8.97 µs	2.18 ms (243x)
select div:has(a) — whatwg spec (235 kB)	6.08 µs	3.16 ms (519x)
find by text content — daring fireball (10 kB)	42.9 µs	55.9 µs (1.4x)
find by text content — ars technica (56 kB)	431 µs	221 µs (0.6x)
find by text content — mozilla blog (95 kB)	329 µs	1.24 ms (3.8x)
find by text content — whatwg spec (235 kB)	638 µs	3.06 ms (4.8x)
collect visible text — daring fireball (10 kB)	2.69 µs	19 µs (7.1x)
collect visible text — ars technica (56 kB)	13.4 µs	76.5 µs (5.8x)
collect visible text — mozilla blog (95 kB)	23.3 µs	166 µs (7.2x)
collect visible text — whatwg spec (235 kB)	76.8 µs	588 µs (7.7x)
serialize a parsed tree — daring fireball (10 kB)	7.14 µs	430 µs (60.4x)
serialize a parsed tree — ars technica (56 kB)	40.7 µs	1.86 ms (45.8x)
serialize a parsed tree — mozilla blog (95 kB)	79.2 µs	4.17 ms (52.7x)
serialize a parsed tree — whatwg spec (235 kB)	209 µs	11.1 ms (53.2x)
tag every link rel=nofollow — daring fireball (10 kB)	3.32 µs	27.3 µs (8.3x)
tag every link rel=nofollow — ars technica (56 kB)	13.2 µs	61.7 µs (4.7x)
tag every link rel=nofollow — mozilla blog (95 kB)	21 µs	128 µs (6.2x)
tag every link rel=nofollow — whatwg spec (235 kB)	42.5 µs	411 µs (9.7x)
class add/remove on every link — daring fireball (10 kB)	2.2 µs	21 µs (9.6x)
class add/remove on every link — ars technica (56 kB)	8.55 µs	69.8 µs (8.2x)
class add/remove on every link — mozilla blog (95 kB)	8.48 µs	129 µs (15.3x)
class add/remove on every link — whatwg spec (235 kB)	8.08 µs	373 µs (46.2x)
drop tags with content (remove) — daring fireball (10 kB)	25.3 µs	1.48 ms (58.4x)
drop tags with content (remove) — ars technica (56 kB)	125 µs	6.52 ms (52.0x)
drop tags with content (remove) — mozilla blog (95 kB)	283 µs	14.7 ms (51.9x)
drop tags with content (remove) — whatwg spec (235 kB)	687 µs	39.7 ms (57.8x)
unwrap tags keep content (strip_tags) — daring fireball (10 kB)	25.8 µs	1.5 ms (58.3x)
unwrap tags keep content (strip_tags) — ars technica (56 kB)	127 µs	6.74 ms (53.0x)
unwrap tags keep content (strip_tags) — mozilla blog (95 kB)	286 µs	14.4 ms (50.5x)
unwrap tags keep content (strip_tags) — whatwg spec (235 kB)	742 µs	39.6 ms (53.4x)
replace body inner HTML — daring fireball (10 kB)	2.33 µs	90.1 µs (38.7x)
replace body inner HTML — ars technica (56 kB)	9.43 µs	145 µs (15.4x)
replace body inner HTML — mozilla blog (95 kB)	15.6 µs	227 µs (14.6x)
replace body inner HTML — whatwg spec (235 kB)	45.4 µs	861 µs (19.0x)
replace body text — daring fireball (10 kB)	1.19 µs	23.9 µs (20.2x)
replace body text — ars technica (56 kB)	7.72 µs	72.4 µs (9.4x)
replace body text — mozilla blog (95 kB)	13 µs	137 µs (10.6x)
replace body text — whatwg spec (235 kB)	39.8 µs	802 µs (20.2x)
walk every descendant — daring fireball (10 kB)	3.34 µs	6.69 µs (2.1x)
walk every descendant — ars technica (56 kB)	13.8 µs	26 µs (1.9x)
walk every descendant — mozilla blog (95 kB)	28.2 µs	54.8 µs (2.0x)
walk every descendant — whatwg spec (235 kB)	101 µs	193 µs (2.0x)
extract @href per match — daring fireball (10 kB)	2.58 µs	99.5 µs (38.6x)
extract @href per match — ars technica (56 kB)	6.65 µs	362 µs (54.5x)
extract @href per match — mozilla blog (95 kB)	9.23 µs	770 µs (83.5x)
extract @href per match — whatwg spec (235 kB)	9.81 µs	2.46 ms (251x)
extract text per match — daring fireball (10 kB)	2.69 µs	108 µs (40.1x)
extract text per match — ars technica (56 kB)	6.68 µs	368 µs (55.2x)
extract text per match — mozilla blog (95 kB)	10.2 µs	1.75 ms (172x)
extract text per match — whatwg spec (235 kB)	10.9 µs	4.32 ms (397x)
detect a byte stream’s encoding — ascii (1 kB)	1.89 µs	1.47 µs (0.8x)
detect a byte stream’s encoding — utf-8 russian (4 kB)	3.64 µs	3.23 µs (0.9x)
detect a byte stream’s encoding — windows-1251 russian (4 kB)	185 µs	3.59 µs (0.1x)
detect a byte stream’s encoding — windows-1252 french (4 kB)	186 µs	3.46 µs (0.1x)
detect a byte stream’s encoding — shift_jis japanese (4 kB)	708 µs	23.6 µs (0.1x)
detect a byte stream’s encoding — utf-8 page (95 kB)	642 ns	15.4 µs (24.0x)

The Performance page benchmarks the build and edit paths against BeautifulSoup too.

How to migrate¶

Swap the import and the constructor. There is no parser name to pass, since turbohtml always runs the WHATWG algorithm:

# BeautifulSoup
from bs4 import BeautifulSoup

soup = BeautifulSoup(markup, "html.parser")

from turbohtml import parse

doc = parse("<p id=intro>Hello</p>")
print(doc.find("p").attrs["id"])

intro

Bytes work too; pass the raw response and read the resolved encoding back from Document.encoding:

doc = parse(b'<meta charset="latin-1"><p>caf\xe9</p>')
print(doc.find("p").text)
print(doc.encoding)  # the WHATWG label latin-1 resolves to

café
windows-1252

API mapping¶

BeautifulSoup	turbohtml
`tag.name`	`tag`
`tag["class"]`, `tag.get("x")`, `tag.has_attr("x")`	`attrs` (`attrs["class"]`, `attrs.get("x")`, `"x" in attrs`)
`tag.string`, `tag.get_text()`	`text`, `strings`, `stripped_strings`
`tag.parents`	`ancestors`
`tag.contents`, `list(tag.children)`	`children`
`tag.next_elements`	`following`
`tag.find_parent(...)`	`find()` (`axis=Axis.ANCESTORS`) or `closest()`
`tag.find_next(...)`, `tag.find_previous(...)`	`find()` with `axis=Axis.FOLLOWING` / `Axis.PRECEDING`
`tag.find_next_sibling(...)`, `tag.find_previous_sibling(...)`	`find()` with `axis=Axis.NEXT_SIBLINGS` / `Axis.PREVIOUS_SIBLINGS`
`tag.find_all("a", recursive=False)`	`find_all()` (`axis=Axis.CHILDREN`)
`soup.find(string=re.compile(r"\$\d+"))`, `soup.find_all(string="Add to cart")`	`find()` / `find_all()` with `text=`
`soup.select(".cls")`, `soup.select_one(".cls")`	`select()`, `select_one()`
(no XPath)	`xpath()`, `xpath_iter()`
`BeautifulSoup(markup, parse_only=SoupStrainer("article"))`	`turbohtml.parse(markup).prune("article")` (`prune()`)
`tag.decompose()`, `tag.extract()`, `tag.unwrap()`, `tag.wrap(...)`	`decompose()`, `extract()`, `unwrap()`, `wrap()`
`tag.insert_before(...)`, `tag.insert_after(...)`, `tag.replace_with(...)`	`insert_before()`, `insert_after()`, `replace_with()`
`soup.new_tag("div")`, `soup.new_string("hi")`	`Element`, `Text`
`tag.prettify()`	`serialize()` (`Html(layout=Indent(2))`)
`tag.smooth()`	`normalize()`
`tag.sourceline`, `tag.sourcepos`	`source_line`, `source_col`

The find/find_all filter grammar covers what bs4 spread across many methods. A keyword filter matches an attribute; class_ and attrs match the rest; axis replaces the directional finders and recursive=False:

from turbohtml import Axis

doc = parse('<ul><li class="x">a</li><li class="y">b</li></ul>')
print([li.text for li in doc.find_all("li")])
print(doc.find("li", class_="y").text)
print(doc.find("ul").find_all("li", axis=Axis.CHILDREN, attrs={"class": "x"}))

['a', 'b']
b
[Element('li')]

Axis reaches every direction a bs4 directional finder did:

deep = parse("<section><p><b>hi</b></p></section>").find("b")
print(deep.find("section", axis=Axis.ANCESTORS).tag)
print(deep.closest("section").tag)

section
section

Gotchas and pitfalls¶

Indexing reaches children, not attributes. node[i] indexes child nodes, so attributes are reached only through .attrs, never node["attr"]. Multi-valued attributes (class, rel, …) read back as a list[str].
```
a = parse('<a class="btn lg" href="/x">go</a>').find("a")
print(a.attrs["class"])
print(a[0]) # indexing reaches children, never attributes
```
```
['btn', 'lg']
Text('go')
```
No ``.string`` shortcut and no ``text``/``tail`` split. Text is real child nodes (the WHATWG DOM shape), so there is no .string and no lxml-style text/tail; iterate the children or read text:
```
p = parse("Hello bold world").find("p")
print((p.text, list(p.stripped_strings)))
```
```
('Hello bold world', ['Hello', 'bold', 'world'])
```
``text=`` filters elements, not ``NavigableString``. bs4’s find(string=...) returns the matching string node; turbohtml’s text= filters elements by their collected subtree text, so it composes with tag and attribute filters. A plain string matches the full text; use a regex to search within:
```
import re

doc = parse("<ul><li>Buy now</li><li>Later</li></ul>")
print(doc.find("li", text="Buy now").text)
print([li.text for li in doc.find_all("li", text=re.compile(r"now"))])
```
```
Buy now
['Buy now']
```

Serialization is WHATWG-conformant by default. Output differs from bs4’s html formatter on named entities, attribute order, and   versus  . Pick Formatter.NAMED_ENTITIES to approximate bs4:

from turbohtml import Formatter, Html

node = parse("<p>café &amp; co</p>").find("p")
print(node.html)
print(node.serialize(Html(formatter=Formatter.NAMED_ENTITIES)))

<p>café &amp; co</p>
<p>caf&eacute; &amp; co</p>

Encoding sniffing stops at the markup. parse runs the WHATWG algorithm on bytes — BOM, then a <meta charset> prescan, then a windows-1252 fallback — which covers what UnicodeDammit reads from the markup but does not guess from byte frequency. A stream with no BOM and no declaration lands on windows-1252 where charset-normalizer might name, say, koi8-r. When there is nothing to sniff, detect first and hand turbohtml the decoded str.
``==`` compares identity; structural equality is a method. Two parses of the same markup are ==-unequal (== stays node identity, so nodes remain usable as dict keys). Where bs4 leaned on structural ==, call equals(): a.equals(b) compares two subtrees by name, attributes, and contents, the bs4 notion of tree equality.
``SoupStrainer`` has no parse-time equivalent. turbohtml always runs the full WHATWG algorithm, then prune() trims the parsed tree to a CSS selector in one C pass, so a large document still yields a small tree — but the whole document is parsed first.