From BeautifulSoup¶

BeautifulSoup is the long-standing convenience layer over a choice of HTML parsers (html.parser, lxml, or html5lib): you pick a backend, then navigate and search the resulting soup with a large, alias-rich API. It also parses XML through the lxml-xml backend, decodes unknown byte streams with UnicodeDammit, and matches CSS selectors through its soupsieve dependency. Its reach and forgiving API made it the default scraping tool for a generation of Python code, from one-off screen scrapes to production pipelines.

turbohtml covers the HTML side of that ground with one engine. turbohtml.parse() runs the WHATWG algorithm in C, returns a fully type annotated Document, and exposes a single find/find_all/select grammar plus XPath. It shares the most surface with turbohtml of any library here, so this is the deepest migration section.

turbohtml vs BeautifulSoup¶

Dimension	turbohtml	BeautifulSoup
Scope	WHATWG HTML: parse, query, mutate, and serialize in one C engine	HTML and XML navigation layer over a pluggable third-party parser
Feature breadth	`find` grammar with `Axis`, CSS selectors, XPath, source positions, linkify, WHATWG serialization formatters	`find` grammar, CSS via `soupsieve`, XML mode, `UnicodeDammit` encoding guessing, output-formatter registry
Performance	C core; one to three orders of magnitude faster than `bs4` over `html.parser` across parse, query, and serialize	Pure-Python navigation; speed tracks the chosen backend and is slowest on `html.parser`
Typing	Ships `py.typed`; every public method annotated	No inline types; relies on the third-party `types-beautifulsoup4` stubs
Dependencies	Zero runtime dependencies (self-contained C extension)	Requires `soupsieve`; `lxml` or `html5lib` optional for those backends
Maintenance	Actively developed, single-engine	Mature, widely deployed, long release history

Feature overlap¶

These port one-to-one; the calls differ only in name (see the mapping table under How to migrate):

Parsing markup into a navigable tree.
find / find_all by tag, attribute, class_, and text.
CSS selectors via select / select_one.
Directional navigation (parents, siblings, next/previous elements) — turbohtml folds these into find with an Axis.
Tree mutation: decompose, extract, unwrap, wrap, insert_before, insert_after, replace_with.
Text access: get_text maps to text, strings, and stripped_strings.
Pretty-printing via serialize() with Html(layout=Indent(2)) for prettify().
Source positions: sourceline / sourcepos map to source_line / source_col.

What turbohtml adds¶

XPath. xpath() and xpath_iter() evaluate expressions with namespaces, variables, and extension functions. BeautifulSoup has no XPath support at all.
A C engine. Parsing, querying, text collection, and serialization all run in C, so migrating off html.parser is a large speedup with no backend to install.
WHATWG-conformant serialization. Formatter selection controls entities, and output matches the spec by default; Formatter.NAMED_ENTITIES reproduces bs4’s html formatter when you need it.
Post-parse pruning. prune() trims a fully parsed tree to a CSS selector in one C pass.
Zero runtime dependencies and full typing. No soupsieve install, and every public API is annotated.

What BeautifulSoup has that turbohtml does not¶

XML parsing. BeautifulSoup(markup, "lxml-xml") parses arbitrary XML. turbohtml runs the WHATWG HTML algorithm only; there is no XML tree-builder. No equivalent — keep lxml for XML documents.
Pluggable parser backends. bs4 lets you swap html.parser, lxml, and html5lib. turbohtml always runs its own WHATWG parser; this is a deliberate clean-break omission, not a gap you work around.
Statistical encoding detection. UnicodeDammit (and chardet / charset-normalizer) guess an encoding from byte frequency when there is no BOM or <meta charset>. turbohtml sniffs only what the WHATWG algorithm reads, then falls back to windows-1252. Workaround: detect with charset-normalizer first and hand turbohtml the decoded str (or bytes with an explicit encoding=).
A named output-formatter registry. bs4 registers custom formatters by name. Workaround: pass a Formatter per serialize() call.

Performance¶

turbohtml parses, queries, and serializes one to three orders of magnitude faster than BeautifulSoup over html.parser; even the closest-margin operations — text filtering (find(text=...) against find_all(string=...)), walking the tree (descendants against soup.descendants), and reading text (text against soup.get_text()) — still run a few times faster:

operation	turbohtml	BeautifulSoup
build a list (constructors) — 100 rows	56.8 µs	756 µs (13.4x)
build a list (constructors) — 1k rows	543 µs	7.39 ms (13.7x)
build a list (constructors) — 10k rows	5.34 ms	80.9 ms (15.2x)
construct N elements (no serialize) — 100 rows	45 µs	262 µs (5.9x)
construct N elements (no serialize) — 1k rows	482 µs	2.55 ms (5.3x)
construct N elements (no serialize) — 10k rows	4.42 ms	25.3 ms (5.8x)
emit a built tree — 100 rows	4.22 µs	389 µs (92.2x)
emit a built tree — 1k rows	42.2 µs	3.89 ms (92.2x)
emit a built tree — 10k rows	420 µs	39.1 ms (93.2x)
parse to a tree — wpt tiny (0.6 kB)	1.21 µs	60.6 µs (50.1x)
parse to a tree — wpt small (4 kB)	9.58 µs	449 µs (46.9x)
parse to a tree — wpt medium (9.6 kB)	24.1 µs	840 µs (35.0x)
parse to a tree — wpt large (92 kB)	209 µs	15.2 ms (72.6x)
parse to a tree — wpt CJK (124 kB)	408 µs	23.3 ms (57.1x)
parse to a tree — whatwg spec (235 kB)	400 µs	25.5 ms (63.8x)
find every anchor — daring fireball (10 kB)	371 ns	13.5 µs (36.4x)
find every anchor — ars technica (56 kB)	817 ns	47.9 µs (58.7x)
find every anchor — mozilla blog (95 kB)	1.16 µs	109 µs (94.0x)
find every anchor — whatwg spec (235 kB)	1.36 µs	350 µs (258x)
select div a[href] — daring fireball (10 kB)	610 ns	169 µs (278x)
select div a[href] — ars technica (56 kB)	1.47 µs	570 µs (390x)
select div a[href] — mozilla blog (95 kB)	2.1 µs	1.08 ms (515x)
select div a[href] — whatwg spec (235 kB)	1.78 µs	2.79 ms (1565x)
select div:has(a) — daring fireball (10 kB)	254 ns	114 µs (449x)
select div:has(a) — ars technica (56 kB)	1.24 µs	493 µs (397x)
select div:has(a) — mozilla blog (95 kB)	8.99 µs	2.24 ms (249x)
select div:has(a) — whatwg spec (235 kB)	5.8 µs	3.22 ms (555x)
match each anchor against div a[href] — daring fireball (10 kB)	1.79 µs	213 µs (120x)
match each anchor against div a[href] — ars technica (56 kB)	4.21 µs	592 µs (141x)
match each anchor against div a[href] — mozilla blog (95 kB)	5.76 µs	876 µs (153x)
match each anchor against div a[href] — whatwg spec (235 kB)	6.68 µs	1.23 ms (185x)
find by text content — daring fireball (10 kB)	24.4 µs	57.7 µs (2.4x)
find by text content — ars technica (56 kB)	178 µs	217 µs (1.3x)
find by text content — mozilla blog (95 kB)	287 µs	463 µs (1.7x)
find by text content — whatwg spec (235 kB)	560 µs	1.67 ms (3.0x)
collect visible text — daring fireball (10 kB)	2.7 µs	18.7 µs (7.0x)
collect visible text — ars technica (56 kB)	13 µs	77.6 µs (6.0x)
collect visible text — mozilla blog (95 kB)	22.4 µs	162 µs (7.3x)
collect visible text — whatwg spec (235 kB)	75.4 µs	566 µs (7.6x)
serialize a parsed tree — daring fireball (10 kB)	7.02 µs	414 µs (59.0x)
serialize a parsed tree — ars technica (56 kB)	38.8 µs	1.82 ms (46.9x)
serialize a parsed tree — mozilla blog (95 kB)	76.3 µs	4.12 ms (54.0x)
serialize a parsed tree — whatwg spec (235 kB)	195 µs	11 ms (56.4x)
class add/remove on every link — daring fireball (10 kB)	2.24 µs	21.3 µs (9.5x)
class add/remove on every link — ars technica (56 kB)	8.89 µs	67.9 µs (7.7x)
class add/remove on every link — mozilla blog (95 kB)	8.78 µs	126 µs (14.4x)
class add/remove on every link — whatwg spec (235 kB)	8.7 µs	381 µs (43.8x)
drop tags with content (remove) — daring fireball (10 kB)	23.8 µs	1.35 ms (56.6x)
drop tags with content (remove) — ars technica (56 kB)	115 µs	5.83 ms (50.7x)
drop tags with content (remove) — mozilla blog (95 kB)	260 µs	13.3 ms (51.4x)
drop tags with content (remove) — whatwg spec (235 kB)	645 µs	38.6 ms (59.9x)
unwrap tags keep content (strip_tags) — daring fireball (10 kB)	24.3 µs	1.42 ms (58.5x)
unwrap tags keep content (strip_tags) — ars technica (56 kB)	122 µs	6.29 ms (51.5x)
unwrap tags keep content (strip_tags) — mozilla blog (95 kB)	264 µs	13.9 ms (52.6x)
unwrap tags keep content (strip_tags) — whatwg spec (235 kB)	683 µs	38 ms (55.6x)
replace body inner HTML — daring fireball (10 kB)	2.12 µs	85.9 µs (40.5x)
replace body inner HTML — ars technica (56 kB)	8.69 µs	132 µs (15.2x)
replace body inner HTML — mozilla blog (95 kB)	14.4 µs	211 µs (14.8x)
replace body inner HTML — whatwg spec (235 kB)	40.1 µs	789 µs (19.7x)
walk every descendant — daring fireball (10 kB)	3.26 µs	6.99 µs (2.2x)
walk every descendant — ars technica (56 kB)	13.4 µs	26 µs (2.0x)
walk every descendant — mozilla blog (95 kB)	28 µs	58 µs (2.1x)
walk every descendant — whatwg spec (235 kB)	96.8 µs	190 µs (2.0x)
extract every link — daring fireball (10 kB)	8.49 µs	15 µs (1.8x)
extract every link — ars technica (56 kB)	27.6 µs	53.1 µs (2.0x)
extract every link — mozilla blog (95 kB)	50.1 µs	117 µs (2.4x)
extract every link — whatwg spec (235 kB)	84.6 µs	369 µs (4.4x)
rewrite every link — daring fireball (10 kB)	4.69 µs	16.5 µs (3.6x)
rewrite every link — ars technica (56 kB)	11.5 µs	56.4 µs (5.0x)
rewrite every link — mozilla blog (95 kB)	21.7 µs	110 µs (5.1x)
rewrite every link — whatwg spec (235 kB)	38.1 µs	351 µs (9.3x)
social-card extraction — head	1.79 µs	68.4 µs (38.2x)
social-card extraction — article 8 KiB	22.1 µs	2.23 ms (101x)
extract @href per match — daring fireball (10 kB)	2.49 µs	93.1 µs (37.5x)
extract @href per match — ars technica (56 kB)	6.29 µs	338 µs (53.8x)
extract @href per match — mozilla blog (95 kB)	8.51 µs	721 µs (84.8x)
extract @href per match — whatwg spec (235 kB)	9.05 µs	2.4 ms (266x)
extract text per match — daring fireball (10 kB)	2.75 µs	110 µs (39.9x)
extract text per match — ars technica (56 kB)	6.39 µs	399 µs (62.5x)
extract text per match — mozilla blog (95 kB)	10.3 µs	779 µs (75.9x)
extract text per match — whatwg spec (235 kB)	10.8 µs	2.92 ms (270x)
extract URL hints — base_url / get_base_url	1.27 µs	53.9 µs (42.5x)
extract URL hints — meta_refresh / get_meta_refresh	1.29 µs	55.1 µs (42.8x)
detect a byte stream’s encoding — ascii (1 kB)	1.96 µs	1.59 µs (0.9x)
detect a byte stream’s encoding — utf-8 russian (4 kB)	3.77 µs	3.29 µs (0.9x)
detect a byte stream’s encoding — windows-1251 russian (4 kB)	206 µs	3.77 µs (0.1x)
detect a byte stream’s encoding — windows-1252 french (4 kB)	206 µs	3.57 µs (0.1x)
detect a byte stream’s encoding — shift_jis japanese (4 kB)	773 µs	24.4 µs (0.1x)
detect a byte stream’s encoding — utf-8 page (95 kB)	705 ns	16.5 µs (23.5x)
extract filtered page links — daring fireball (10 kB)	127 µs	19.4 µs (0.2x)
extract filtered page links — ars technica (56 kB)	310 µs	59.6 µs (0.2x)
extract filtered page links — mozilla blog (95 kB)	514 µs	119 µs (0.3x)
extract filtered page links — whatwg spec (235 kB)	889 µs	395 µs (0.5x)

The Performance page benchmarks the build and edit paths against BeautifulSoup too.

How to migrate¶

Swap the import and the constructor. There is no parser name to pass, since turbohtml always runs the WHATWG algorithm:

# BeautifulSoup
from bs4 import BeautifulSoup

soup = BeautifulSoup(markup, "html.parser")

from turbohtml import parse

doc = parse("<p id=intro>Hello</p>")
print(doc.find("p").attrs["id"])

intro

Bytes work too; pass the raw response and read the resolved encoding back from Document.encoding:

doc = parse(b'<meta charset="latin1"><p>caf\xe9</p>')
print(doc.find("p").text)
print(doc.encoding)  # the encoding the WHATWG label latin1 names

café
windows-1252

API mapping¶

BeautifulSoup	turbohtml
`tag.name`	`tag`
`tag["class"]`, `tag.get("x")`, `tag.has_attr("x")`	`attrs` (`attrs["class"]`, `attrs.get("x")`, `"x" in attrs`)
`tag.string`, `tag.get_text()`	`text`, `strings`, `stripped_strings`
`tag.parents`	`ancestors`
`tag.contents`, `list(tag.children)`	`children`
`tag.next_elements`	`following`
`tag.find_parent(...)`	`find()` (`axis=Axis.ANCESTORS`) or `closest()`
`tag.find_next(...)`, `tag.find_previous(...)`	`find()` with `axis=Axis.FOLLOWING` / `Axis.PRECEDING`
`tag.find_next_sibling(...)`, `tag.find_previous_sibling(...)`	`find()` with `axis=Axis.NEXT_SIBLINGS` / `Axis.PREVIOUS_SIBLINGS`
`tag.find_all("a", recursive=False)`	`find_all()` (`axis=Axis.CHILDREN`)
`soup.find(string=re.compile(r"\$\d+"))`, `soup.find_all(string="Add to cart")`	`find()` / `find_all()` with `text=`
`soup.select(".cls")`, `soup.select_one(".cls")`	`select()`, `select_one()`
(no XPath)	`xpath()`, `xpath_iter()`
`BeautifulSoup(markup, parse_only=SoupStrainer("article"))`	`turbohtml.parse(markup).prune("article")` (`prune()`)
`tag.decompose()`, `tag.extract()`, `tag.unwrap()`, `tag.wrap(...)`	`decompose()`, `extract()`, `unwrap()`, `wrap()`
`tag.insert_before(...)`, `tag.insert_after(...)`, `tag.replace_with(...)`	`insert_before()`, `insert_after()`, `replace_with()`
`soup.new_tag("div")`, `soup.new_string("hi")`	`Element`, `Text`
`tag.prettify()`	`serialize()` (`Html(layout=Indent(2))`)
`tag.smooth()`	`normalize()`
`tag.sourceline`, `tag.sourcepos`	`source_line`, `source_col`

The find/find_all filter grammar covers what bs4 spread across many methods. A keyword filter matches an attribute; class_ and attrs match the rest; axis replaces the directional finders and recursive=False:

from turbohtml import Axis

doc = parse('<ul><li class="x">a</li><li class="y">b</li></ul>')
print([li.text for li in doc.find_all("li")])
print(doc.find("li", class_="y").text)
print(doc.find("ul").find_all("li", axis=Axis.CHILDREN, attrs={"class": "x"}))

['a', 'b']
b
[Element('li')]

Axis reaches every direction a bs4 directional finder did:

deep = parse("<section><p><b>hi</b></p></section>").find("b")
print(deep.find("section", axis=Axis.ANCESTORS).tag)
print(deep.closest("section").tag)

section
section

Gotchas and pitfalls¶

Indexing reaches children, not attributes. node[i] indexes child nodes, so attributes are reached only through .attrs, never node["attr"]. Multi-valued attributes (class, rel, …) read back as a list[str].
```
a = parse('<a class="btn lg" href="/x">go</a>').find("a")
print(a.attrs["class"])
print(a[0]) # indexing reaches children, never attributes
```
```
['btn', 'lg']
Text('go')
```
No ``.string`` shortcut and no ``text``/``tail`` split. Text is real child nodes (the WHATWG DOM shape), so there is no .string and no lxml-style text/tail; iterate the children or read text:
```
p = parse("Hello bold world").find("p")
print((p.text, list(p.stripped_strings)))
```
```
('Hello bold world', ['Hello', 'bold', 'world'])
```
``text=`` filters elements, not ``NavigableString``. bs4’s find(string=...) returns the matching string node; turbohtml’s text= filters elements by their collected subtree text, so it composes with tag and attribute filters. A plain string matches the full text; use a regex to search within:
```
import re

doc = parse("<ul><li>Buy now</li><li>Later</li></ul>")
print(doc.find("li", text="Buy now").text)
print([li.text for li in doc.find_all("li", text=re.compile(r"now"))])
```
```
Buy now
['Buy now']
```

Serialization is WHATWG-conformant by default. Output differs from bs4’s html formatter on named entities, attribute order, and   versus  . Pick Formatter.NAMED_ENTITIES to approximate bs4:

from turbohtml import Formatter, Html

node = parse("<p>café &amp; co</p>").find("p")
print(node.html)
print(node.serialize(Html(formatter=Formatter.NAMED_ENTITIES)))

<p>café &amp; co</p>
<p>caf&eacute; &amp; co</p>

Encoding sniffing stops at the markup. parse runs the WHATWG algorithm on bytes — BOM, then a <meta charset> prescan, then a windows-1252 fallback — which covers what UnicodeDammit reads from the markup but does not guess from byte frequency. A stream with no BOM and no declaration lands on windows-1252 where charset-normalizer might name, say, koi8-r. When there is nothing to sniff, detect first and hand turbohtml the decoded str.
``==`` compares identity; structural equality is a method. Two parses of the same markup are ==-unequal (== stays node identity, so nodes remain usable as dict keys). Where bs4 leaned on structural ==, call equals(): a.equals(b) compares two subtrees by name, attributes, and contents, the bs4 notion of tree equality.
``SoupStrainer`` has no parse-time equivalent. turbohtml always runs the full WHATWG algorithm, then prune() trims the parsed tree to a CSS selector in one C pass, so a large document still yields a small tree — but the whole document is parsed first.