From metadata_parser¶
metadata_parser reads the social-card metadata a page advertises: the
OpenGraph (og:) and Twitter (twitter:) <meta> tags, plus Dublin Core (dc), plain name/content
pairs, and page-level links such as canonical and shortlink. It builds a MetadataParser over the HTML (or
fetches it for you from a URL), parses with BeautifulSoup, and groups the tags into per-namespace dicts on
parsed_result.metadata (["og"], ["twitter"], ["dc"], ["meta"], ["page"]) that you read by key or
through get_metadata/get_metadata_link with a namespace strategy. Its extra weight is URL work: it fetches
pages, follows redirects, and resolves relative URL-valued tags against the page URL to pick a single “discrete” URL for
sharing.
turbohtml covers the social-card ground with opengraph(), which gathers the og: and
twitter: <meta> tags into one flat {key: value} mapping in a single C walk over the WHATWG-parsed tree, with
no parser object to construct and no per-namespace accessor to pick. It parses the markup you already have rather than
fetching, so it slots behind whatever downloader you use.
turbohtml vs metadata_parser¶
Dimension |
turbohtml |
metadata_parser |
|---|---|---|
Scope |
WHATWG HTML parser with metadata extraction on parsed trees |
Social-card metadata reader with URL fetching and normalization |
Feature breadth |
Full DOM, selectors, serialization, |
|
Performance |
One C walk over the parsed tree (see below) |
BeautifulSoup tree build and Python namespace mapping |
Typing |
Fully typed, ships stubs |
No published type stubs |
Dependencies |
Zero runtime dependencies (native C extension) |
requests, BeautifulSoup4, and helper libraries |
Maintenance |
Actively maintained |
Actively maintained |
Feature overlap¶
Port these 1:1:
mp.parsed_result.metadata["og"]["title"]-> theog:titlekey ofopengraph().mp.parsed_result.get_metadatas("card", strategy=["twitter"])-> thetwitter:cardkey ofopengraph().mp.parsed_result.metadata["twitter"]["image"]-> thetwitter:imagekey of the same mapping (og:andtwitter:tags share the one dict).MetadataParser(html=html)->parse(html)(turbohtml.parse()), then readopengraph().
What turbohtml adds¶
A full WHATWG-conformant parser: the tree
opengraph()walks is the same DOM you query with CSS selectors, serialize, or mutate, not a metadata-private intermediate.Zero runtime dependencies. metadata_parser pulls in requests and BeautifulSoup4; turbohtml is a self-contained C extension.
Full type coverage with shipped stubs, so
opengraph()and its result check under a type checker.Structured data beyond the social card:
structured_data()returns JSON-LD, Microdata, RDFa, and Dublin Core in the same walk, withjson_ld(),microdata(),rdfa(), anddublin_core()besideopengraph().Document URL hints on the parsed tree:
base_url()resolves the effective<base>andmeta_refresh()reads the refresh target.A standalone string entry point
turbohtml.extract.opengraph()that strips theog:prefix and addsis_valid()for the Open Graph protocol’s required properties.
What metadata_parser has that turbohtml does not¶
URL fetching:
MetadataParser(url=...)downloads the page, follows redirects, and detects the response encoding. turbohtml takes parsed HTML only, so pair it withurlliborhttpx.Generic ``name``/``content`` namespace: metadata_parser groups arbitrary
<meta name>tags intometadata["meta"].opengraph()gathers onlyog:/twitter:tags anddublin_core()thedc.*/dcterms.*ones; read any other<meta>by selecting it, e.g.parse(html).find_all("meta").Page-level links collected into
metadata["page"]. The canonical URL comes back througharticle()asarticle().canonical(the<link rel="canonical">href, falling back toog:url); read any other page link, such asshortlink, with a selector, e.g.parse(html).find('link[rel="shortlink"]').Namespace ``strategy`` ordering:
get_metadata(field, strategy=[...])picks the first namespace that carries a field. turbohtml keys tags by their full prefix (og:titlevstwitter:title), so read the prefix you want directly.
Performance¶
Both libraries start from the raw HTML string, so each parses before it reads the tags: metadata_parser builds its
own tree and maps the meta block in Python, where opengraph() parses to the WHATWG tree and
gathers the og:/twitter: tags in one C walk. On a social-card head, and on an 8 KiB article carrying that head,
the single pass runs dozens of times faster:
input |
turbohtml |
|
|---|---|---|
head |
1.9 µs |
101 µs (52.9x) |
article 8 KiB |
25.3 µs |
2.42 ms (95.7x) |
How to migrate¶
Swap the MetadataParser construction for a turbohtml.parse() call, then read
opengraph() by prefixed key instead of picking a namespace dict. Where metadata_parser fetched
the URL for you, download the markup first and pass it to turbohtml.parse().
turbohtml |
|
|---|---|
|
|
|
the |
|
the |
|
the |
|
|
|
|
from turbohtml import parse
doc = parse('<head><meta property="og:title" content="Widget"><meta name="twitter:card" content="summary"></head>')
print(doc.structured_data().opengraph)
{'og:title': 'Widget', 'twitter:card': 'summary'}
Gotchas and pitfalls¶
metadata_parser groups tags into per-namespace dicts with unprefixed keys (
metadata["og"]["title"]), whileopengraph()returns one flat mapping with prefixed keys (og:title,twitter:card) because pages mix thepropertyandnameattributes freely.opengraph()returns the raw tag value by default; passbase_url=(the page URL) to absolutize the URL-valued tags (og:url,og:image,og:video,twitter:image, …) against it, the waymetadata_parser’sget_metadata_linkandget_discrete_urldo. A<base href>refines the base URL.When a key repeats, the last occurrence wins in the flat mapping; read
structured_data()when you need every occurrence of a repeated key.metadata_parser can fetch (
MetadataParser(url=...)) and detect the response encoding; turbohtml takes parsed HTML, so fetch the page yourself and pass the markup toturbohtml.parse().Only
og:andtwitter:tags land inopengraph(); Dublin Core comes back throughdublin_core(). Genericname/contentpairs and page-level<link>tags come back through selectors, not the metadata mapping.