From MechanicalSoup

MechanicalSoup is a stateful headless browser for Python. It wires together three libraries: it fetches pages with requests, parses them with BeautifulSoup, and adds a StatefulBrowser that remembers the current page and form so you can follow links, fill in fields, and submit without hand-assembling each request. It is built for scripting sites that have no API – logging in, walking paginated results, posting a form – where you want requests-level control but not a full JavaScript engine like Selenium or Playwright.

turbohtml replaces the parsing and form-reading half of that stack. Once a page is fetched, turbohtml parses it with a WHATWG-conformant C parser and exposes the form controls through typed methods, so select_form, reading and setting fields, and collecting the pairs to submit all become calls on the parsed tree. The HTTP session, navigation, and the actual POST stay out of scope: you drive those with requests or httpx and hand turbohtml’s output to your client.

turbohtml vs MechanicalSoup

Dimension	turbohtml	MechanicalSoup
Scope	HTML parsing, CSS queries, and typed form-control reading/writing on the parsed tree	Stateful browser: HTTP session, link/form navigation, and submission on top of requests + BeautifulSoup
Feature breadth	Spec parser, CSS/XPath selectors, serialization, typed form data; no HTTP or session state	StatefulBrowser with cookies, history, form auto-fill and submit; parsing delegated to BeautifulSoup
Performance	WHATWG parser in C; see below	Bounded by requests I/O and BeautifulSoup’s Python parsing
Typing	Ships type stubs; field_value, checked, and form_data are typed	No bundled type stubs
Dependencies	Self-contained C extension, no runtime deps	Depends on requests and beautifulsoup4 (plus a BeautifulSoup parser backend)
Maintenance	Actively developed alongside the turbohtml parser	Actively maintained, thin layer over its two dependencies

Feature overlap

The surface you can port 1:1 is the form-reading and page-querying half:

• Selecting a form on the page: browser.select_form("form") -> select_one() or find().

• Reading a control’s value: form[name] -> field_value.

• Setting a control’s value: browser[name] = value / form.set(name, value) -> field_value assignment.

• Checkbox and radio state: form.check(name) -> checked.

• The name/value pairs a submit would send: what browser.submit_selected() posts -> form_data().

What turbohtml adds

• A WHATWG-conformant HTML parser in C rather than BeautifulSoup’s Python tree builder, so malformed markup is fixed up the way a browser would.

• form_data() applies the WHATWG form-submission rules directly: unnamed, disabled, and button controls are skipped, unchecked checkboxes and radios contribute nothing, and a select yields one pair per selected option. MechanicalSoup reproduces this by walking the BeautifulSoup tree itself.

• Full CSS selector support plus XPath for locating forms and fields, not just tag/attribute lookups.

• Bundled type stubs for the form API.

• No runtime dependencies: turbohtml is a single C extension, where MechanicalSoup pulls in requests and beautifulsoup4.

What MechanicalSoup has that turbohtml does not

• Stateful browsing. StatefulBrowser keeps the current page, form, cookies, and history, and open, follow_link, and submit_selected drive them. turbohtml has no HTTP or session layer. Workaround: fetch with requests or httpx (which manage cookies and redirects for you) and parse each response with turbohtml.

• One-call form submission. submit_selected reads the form, builds the request, and posts it in a single call, honoring the form’s action and method. turbohtml gives you the data via form_data(); you POST it yourself. No equivalent one-call submit.

• Link and page navigation helpers. links(), follow_link, and get_current_page operate on browser state. turbohtml can find the same anchors with select(), but resolving them against the current URL and fetching is your code’s job.

• ``launch_browser`` to open the current page in a real browser for debugging has no turbohtml equivalent.

Performance

Not directly benchmarked.

How to migrate

Swap the import and let your HTTP client stay separate from parsing:

import mechanicalsoup

browser = mechanicalsoup.StatefulBrowser()

becomes

import requests
from turbohtml import parse

API mapping:

MechanicalSoup	turbohtml
browser.open(url), browser.get_current_page()	fetch yourself, then turbohtml.parse(response.content)
browser.select_form("form")	doc.select_one("form")
browser[name] = value, form.set(name, value)	field.field_value = value
reading form[name]	field.field_value
form.check(name) / checkbox & radio state	field.checked = True
the data browser.submit_selected() posts	form.form_data() (hand to your HTTP client)

Before and after, reading a value and collecting the submission pairs:

form = parse(
    "<form><input name=email value=a@b.c>"
    "<select name=plan><option value=free>Free<option value=pro selected>Pro</select>"
    "<input name=terms type=checkbox value=yes></form>"
).find("form")
form.find("input", attrs={"name": "terms"}).checked = True
print(form.find("input", attrs={"name": "email"}).field_value)
print(form.form_data())

a@b.c
[('email', 'a@b.c'), ('plan', 'pro'), ('terms', 'yes')]

The list form_data() returns drops straight into urllib.parse.urlencode() or a requests/httpx data= argument, so submitting is one call on your own client:

import requests

response = requests.post(action_url, data=form.form_data(), timeout=30)

Gotchas and pitfalls

• No session state. MechanicalSoup carried cookies and the current page across open/submit calls. With turbohtml you reuse a requests.Session (or an httpx.Client) for cookies and redirects, and re-parse each response; turbohtml holds no navigation state.

• Submission is spec-driven, not a tree walk. form_data() follows the WHATWG rules: unnamed, disabled, and button controls are skipped, unchecked checkbox/radio controls contribute nothing, and a select produces one pair per selected option. If you relied on MechanicalSoup including a differently-shaped set of controls, compare the form_data output against what you posted before.

• Selector reaches the wrong form on multi-form pages. select_one("form") returns the first match. On a page with several forms, pass a specific selector (an id, name, or action attribute) to reach the one you mean, the same way you would target a form in the browser.

• Parse bytes, not text. Hand turbohtml.parse() the response content (bytes) so turbohtml applies WHATWG encoding detection, rather than pre-decoding to str with a guessed codec.