From microdata¶

microdata is a small pure-Python library that pulls the HTML Microdata items a page embeds – the WHATWG itemscope/itemprop vocabulary sites use to mark up schema.org products, recipes, people, and events. Its get_items entry point parses the page with html5lib, walks the itemscope elements in Python, and returns Item objects you read with .itemtype, .itemid, .props, .get/.get_all, and .json. It has long been the go-to dependency for scrapers that need schema.org markup off a page.

turbohtml serves that surface from turbohtml.extract.microdata(), backed by the same C walk that turbohtml.Document.microdata() runs over the WHATWG tree. It returns the page’s top-level items as frozen MicrodataItem records with the same accessors, so a scraper’s item-reading code ports without a rewrite while the parse-and-walk runs in the C extension rather than in Python over an html5lib tree.

turbohtml vs microdata¶

Dimension	turbohtml	microdata
Scope	Full WHATWG HTML5 parser, serializer, and an extraction suite (Microdata, JSON-LD, OpenGraph, links, dates, boilerplate)	Microdata extraction only
Feature breadth	Microdata, JSON-LD, and OpenGraph gathered in one walk; nested `itemscope` returns as a nested record	`get_items` from a URL, file, or string; nested items; a mutable `Item` you can also build and edit
Performance	One C walk over the parsed tree; 35 to 42x faster on the benchmark pages	Python walk over an html5lib tree, built on each `get_items` call
Typing	Fully typed; `MicrodataItem` is a frozen, slotted dataclass with shipped stubs	Untyped
Dependencies	C extension, no pure-Python runtime dependencies	Requires html5lib
Maintenance	Actively developed	Minimal, low-activity single-maintainer project, stable for years

Feature overlap¶

The item-reading surface ports one-to-one:

microdata.get_items(html) -> turbohtml.extract.microdata(), both returning the page’s top-level items in document order.
item.itemid -> item.id.
item.props -> item.properties, each itemprop name mapped to its values in document order.
item.get(name) / item.get_all(name) -> get() / get_all(), with the same first-value-or-None and list-or-[] contract.
item.json() -> json(), the same two-space-indented JSON of the tree below the item.
A property that is itself an itemscope comes back as a nested item in both libraries, not a flat string.

What turbohtml adds¶

Nested items and multi-valued properties resolve directly off the record, so a property that is another itemscope is a nested MicrodataItem:

from turbohtml.extract import microdata

item = microdata(
    '<div itemscope itemtype="https://schema.org/Person">'
    '<span itemprop="name">Ada</span>'
    '<span itemprop="name">Lovelace</span>'
    '<div itemprop="address" itemscope><span itemprop="city">London</span></div>'
    "</div>"
)[0]
print(item.type)
print(item.get("name"), item.get_all("name"))
print(item.get("address").get("city"))

https://schema.org/Person
Ada ['Ada', 'Lovelace']
London

JSON-LD and OpenGraph off the same page in the same C walk: turbohtml.Document.json_ld(), turbohtml.Document.opengraph(), and turbohtml.Document.structured_data(), plus the string-input helpers turbohtml.extract.opengraph() and the rest of the extraction suite. microdata covers Microdata alone.
turbohtml.Document.microdata() extracts from an already-parsed turbohtml.parse() document, so pages you hold for other work do not get reparsed.

The records are frozen and hold no reference back into the tree, so they outlive the document:

item = microdata('<div itemscope itemid="urn:isbn:1"><span itemprop="name">B</span></div>')[0]
print(item.id, item.json())

urn:isbn:1 {
  "id": "urn:isbn:1",
  "properties": {
    "name": [
      "B"
    ]
  }
}

A text property’s value follows the spec’s textContent rule, so text nested in the property element is kept where microdata drops it (see Gotchas and pitfalls).
URL-valued properties (an a/area/link href, a media src, an object data) are returned verbatim by default; pass base_url= to microdata() (or microdata()) to absolutize them against it, the way microdata’s url argument resolves them. A <base href> refines the base URL.

What microdata has that turbohtml does not¶

Fetching from a URL or file. get_items(location, encoding=None) accepts a URL or file-like object and reads it for you. microdata() takes an HTML string only. Workaround: fetch the bytes yourself (urllib/requests) and pass the decoded text, or feed the response to turbohtml.parse() and call microdata().
An explicit encoding argument. get_items takes encoding to decode the fetched bytes. turbohtml handles encoding at parse time on the byte input instead; there is no separate argument on the string helper.
``itemtype`` as parsed URI objects. item.itemtype is a list of URI objects (item.itemtype[0].string). type is the raw itemtype string; call .type.split() for the token list.
A mutable, buildable item. microdata’s Item is an open object whose props dict you can edit and add to. MicrodataItem is a frozen read-only record. No equivalent; build your own structure if you need to synthesize or mutate items.

Performance¶

microdata starts from the raw HTML string, so it parses before it extracts: it builds an html5lib tree and walks the itemscope elements in Python, where microdata() parses to the WHATWG tree and gathers the items in one C walk. On a product page carrying Microdata alongside JSON-LD and OpenGraph, and on an 8 KiB catalog of the same, the walk runs 35 to 42 times faster:

structured-data extraction	turbohtml	microdata
product	6.22 µs	243 µs (39.1x)
catalog 8 KiB	56.3 µs	2.58 ms (45.8x)

Over the fixtures in microdata’s own test suite the two libraries return byte-identical json() for every page but one, where turbohtml follows the spec more closely (see Gotchas and pitfalls).

How to migrate¶

Swap the import and the entry point:

# before
from microdata import get_items

# after
from turbohtml.extract import microdata

The call and item API map directly:

microdata	turbohtml
`microdata.get_items(html)`	`turbohtml.extract.microdata()`
`item.itemtype` (a list of `URI`)	`item.type` (the `itemtype` string verbatim)
`item.itemid`	`item.id`
`item.props`	`item.properties`
`item.get(name)` / `item.get_all(name)`	`get()` / `get_all()`
`item.json()`	`json()`

The item-reading code is the same shape on both sides; only the itemtype accessor changes:

# before
from microdata import get_items

items = get_items(
    '<div itemscope itemtype="https://schema.org/Person">'
    '<span itemprop="name">Ada</span>'
    '<div itemprop="address" itemscope><span itemprop="city">London</span></div>'
    "</div>"
)
item = items[0]
print(item.itemtype[0].string)
print(item.get("name"))
print(item.get("address").get("city"))

# after
from turbohtml.extract import microdata

item = microdata(
    '<div itemscope itemtype="https://schema.org/Person">'
    '<span itemprop="name">Ada</span>'
    '<div itemprop="address" itemscope><span itemprop="city">London</span></div>'
    "</div>"
)[0]
print(item.type)
print(item.get("name"))
print(item.get("address").get("city"))

Gotchas and pitfalls¶

item.itemtype is a list of URI objects, read via item.itemtype[0].string. type is the raw itemtype string, which the WHATWG spec treats as a space-separated token set. Call .type.split() for the list, the form json() emits.
A text property’s value is the element’s textContent per the spec, so a <script> or <style> nested inside a property element contributes its text; microdata drops that text. This is the one page in microdata’s fixtures where the json() output differs.
get() returns the first value or None and get_all() returns the list or [], matching Item.get/Item.get_all; a property that is itself an itemscope comes back as a nested MicrodataItem, not a string.
microdata() takes an HTML string and reparses it on each call. To read a URL or file, fetch the text yourself first. When you also need json_ld() or opengraph() off the same page, hold a turbohtml.parse() document and call microdata() so the page is parsed once.