Structured-data extraction

structured_data() answers the scraping question extruct and metadata_parser exist for: what machine-readable metadata does this page embed? It pulls JSON-LD, Microdata, OpenGraph/Twitter card metadata, RDFa, and Dublin Core in one walk, with the per-format helpers json_ld(), opengraph(), microdata(), rdfa(), and dublin_core() beside it. The combined result is a frozen StructuredData record with a stable six-field shape – json_ld, microdata, opengraph, rdfa, dublin_core, plus microformats reserved for a later phase – so code that reads it does not break when that format lands. The records hold no reference back into the tree, so they outlive the document they came from.

Where the work runs

The division of labor is the same one the rest of the read path follows: the locating runs in C and the only genuinely Python step stays in Python. A pure-C pass under the per-tree critical section walks the document once per format, gathering the itemscope/itemprop/itemtype structure into nested MicrodataItem records and the OpenGraph and Twitter <meta> pairs into a flat mapping, all holding no reference back into the tree. JSON-LD is the one exception: the C walk gathers the verbatim text of each <script type="application/ld+json"> block into a list of strings, then the critical section is released and a thin facade parses them with the standard library json. The JSON grammar is not reinvented in C, and the parse never touches the tree, so it cannot race a concurrent mutation – the snapshot-then-parse split is what keeps the Python call off the live structure. A block that is not valid JSON is skipped rather than raising, the safe default for scraping a page whose markup the author did not validate.

Microdata, OpenGraph, Twitter

Microdata follows the HTML value algorithm: a property’s value is the nested item dict when the element carries itemscope, otherwise its content for a meta element, the relevant URL attribute for the link and media tags, datetime (or text) for a time element, and the element’s text content elsewhere. Only top-level items – an element with itemscope and no itemprop, so it is not a property of another item – become list entries; a nested item is reached through its parent’s properties. OpenGraph and Twitter share one mapping because pages mix the property and name attributes freely; when a key repeats, the last occurrence wins.

RDFa, Dublin Core

RDFa reuses the Microdata item shape instead of emitting raw triples: a typeof element becomes a RdfaItem, with typeof playing itemscope’s role, property playing itemprop’s, resource/about the subject, and a nested typeof the nested item. The RDFa-specific step is term expansion. A bare property/typeof token joins the in-scope @vocab, and a prefix:reference token joins the prefix’s IRI from the in-scope @prefix map, which starts from the RDFa 1.1 initial context so schema:, dc:, and foaf: resolve without a page-level declaration. @vocab and @prefix thread down the tree and can be overridden per subtree (an empty @vocab clears it); a token whose prefix is undeclared, or a bare term with no vocabulary in scope, stays verbatim. The object of a property follows the RDFa rules: the content literal, else a nested item, else the resource/href/src IRI, else a <time> datetime, else the text. turbohtml does not expand CURIEs embedded in arbitrary IRIs or surface the datatype type IRI, so typed literals come back as their lexical string. Dublin Core is a flat mapping like OpenGraph: the dc.*/dcterms.* <meta> names, lower-cased so DC.Title and dc.title key alike, last occurrence winning. Microformats2 remains a later phase.