Structured-data extraction¶
structured_data() answers the scraping question extruct and metadata_parser exist for:
what machine-readable metadata does this page embed? It pulls JSON-LD, Microdata, OpenGraph/Twitter card metadata, RDFa,
and Dublin Core in one call, with the per-format helpers json_ld(),
opengraph(), microdata(), rdfa(), and
dublin_core() beside it. The combined result is a frozen StructuredData
record with a stable six-field shape – json_ld, microdata, opengraph, rdfa, dublin_core, plus
microformats reserved for a later phase – so code that reads it does not break when that format lands. The records
hold no reference back into the tree, so they outlive the document they came from.
Nested Microdata and RDFa resources become nested Python records. The extractor raises RecursionError when a
record contains more than 400 nested records, or when Microdata itemref links form a nested-item cycle. DOM wrappers
do not count toward that limit, and the flat metadata helpers remain iterative.
Where the work runs¶
C locates and assembles the values; Python parses JSON. Each per-format helper walks the document under its tree lock.
structured_data() copies the document once under that lock, then runs every format pass
against the private copy. Each result comes from one document version even when another thread mutates the original.
JSON-LD uses the same boundary: C gathers the verbatim text of each <script type="application/ld+json"> block, then
a thin facade parses the strings with json without touching the live tree. A block that is not valid JSON is
skipped so extraction can continue when a page contains malformed metadata.
Microdata, OpenGraph, Twitter¶
Microdata follows the HTML value algorithm: a property’s value is the nested item dict when the element carries
itemscope, otherwise its content for a meta element, the relevant URL attribute for the link and media tags,
datetime (or text) for a time element, and the element’s text content elsewhere. Only top-level items – an element
with itemscope and no itemprop, so it is not a property of another item – become list entries; a nested item is
reached through its parent’s properties. OpenGraph and Twitter share one mapping because pages mix the property
and name attributes freely; when a key repeats, the last occurrence wins.
RDFa, Dublin Core¶
RDFa reuses the Microdata item shape instead of emitting raw triples: a typeof element becomes a
RdfaItem, with typeof playing itemscope’s role, property playing itemprop’s,
resource/about the subject, and a nested typeof the nested item. The RDFa-specific step is term expansion. A
bare property/typeof token joins the in-scope @vocab, and a prefix:reference token joins the prefix’s
IRI from the in-scope @prefix map, which starts from the RDFa 1.1 initial context so schema:, dc:, and
foaf: resolve without a page-level declaration. @vocab and @prefix thread down the tree and can be
overridden per subtree (an empty @vocab clears it); a token whose prefix is undeclared, or a bare term with no
vocabulary in scope, stays verbatim. The object of a property follows the RDFa rules: the content literal, else a
nested item, else the resource/href/src IRI, else a <time> datetime, else the text. turbohtml does
not expand CURIEs embedded in arbitrary IRIs or surface the datatype type IRI, so typed literals come back as their
lexical string. Dublin Core is a flat mapping like OpenGraph: the dc.*/dcterms.* <meta> names, lower-cased
so DC.Title and dc.title key alike, last occurrence winning. Microformats2 remains a later phase.