Extract

Pull content and data out of HTML. Extraction runs through the node methods, and the records those methods return are re-exported from this namespace for discoverability – Article, Link, StructuredData, and MicrodataItem – while staying importable from the package root.

boilerplate() adds the per-paragraph view of the main-content scoring: each paragraph unit of a page classified good or boilerplate, the justext / boilerpy3 call shape.

turbohtml.extract.boilerplate(html, options=None, /)

Segment a page into paragraph units and classify each as content or boilerplate, in document order.

The successor to justext.justext and boilerpy3’s per-block is_content: the C main-content scoring picks the content body exactly as turbohtml.Node.main_content() does, every unit outside it is boilerplate, and a unit inside it must still clear the options length and link-density thresholds. A page with no scoring content body (a stub, pure navigation) classifies every unit as boilerplate.

Parameters:
Return type:

list[Paragraph]

Returns:

one Paragraph per non-blank unit.

class turbohtml.extract.Paragraph(text: str, is_boilerplate: bool, is_heading: bool)

One paragraph unit of a page and its classification, the record boilerplate() returns.

text is the unit’s whitespace-normalized visible text (never empty; a blank unit is not reported). is_boilerplate is True for navigation, footers, sidebars, and any in-content unit that fails the Extraction thresholds; the False entries concatenate to the article. is_heading marks <h1>-<h6> units whatever their classification.

Parameters:
class turbohtml.extract.Extraction(min_length=25, max_link_density=0.5, keep_headings=True)

Options for boilerplate(): the per-paragraph thresholds refining the main-content scoring.

Parameters:

  • min_length (int) – a non-heading paragraph inside the content body shorter than this many normalized characters is still boilerplate. The default 25 is the floor the C scoring itself uses for a paragraph to contribute.

  • max_link_density (float) – a paragraph whose text sits inside links beyond this fraction is boilerplate wherever it lives; 0.5 tolerates prose with citations while dropping link lists that survive inside the content body.

  • keep_headings (bool) – with True a heading inside the content body is content regardless of min_length (headings are short by nature); with False headings face the same length threshold as prose, justext’s no_headings mode.

classmethod justext()

Justext’s defaults: its 70-character length_low floor and 0.2 max_link_density.

Return type:

Extraction

dates() recovers a page’s publication or modification date from its <meta> tags, JSON-LD, <time> elements, and URL, the standalone entry point htmldate exposes (htmldate guide). It returns a PublicationDate naming the signal the date came from, configured by a frozen DateExtraction.

turbohtml.extract.dates(html, options=None, /)

Find a document’s publication (or, by default, modification) date, the htmldate.find_date counterpart.

The signals are tried in htmldate’s order – a date in the canonical URL, then publication/modification <meta> tags, then JSON-LD, then <time> elements, then (with extensive_search) visible text – and the first that yields a date inside the [min_date, max_date] window wins. Within a stage the original flag picks a publication date over a modification date, or the reverse.

Parameters:

  • html (str) – the page markup.

  • options (DateExtraction | None) – the extraction knobs; defaults to DateExtraction (modification date, ISO output, the 1995-to-today window, text search on).

Return type:

PublicationDate | None

Returns:

the date and the signal it came from, or None when no bounded date is found.

class turbohtml.extract.PublicationDate(date: str, signal: Signal)

A date dates() recovered and the signal it read it from.

date is the formatted string (output_format, ISO YYYY-MM-DD by default). signal names the engine: "meta" a publication/modification <meta> tag, "json-ld" a JSON-LD datePublished/dateModified, "time" a <time> element, "url" a date pattern in the canonical URL, "text" visible page text.

Parameters:

  • date (str)

  • signal (Literal['url', 'meta', 'json-ld', 'time', 'text'])

class turbohtml.extract.DateExtraction(original=False, output_format='%Y-%m-%d', min_date=None, max_date=None, extensive_search=True)

Options for dates(), mirroring htmldate.find_date’s knobs.

Parameters:

  • original (bool) – prefer the first-published date over the last-modified one, htmldate’s original_date. The default (False) prefers the modification date, the most recent the page reports.

  • output_format (str) – an strftime() format for the returned string; the default is ISO %Y-%m-%d.

  • min_date (date | None) – the earliest acceptable date; a candidate before it is skipped. Defaults to 1995-01-01, the floor htmldate uses.

  • max_date (date | None) – the latest acceptable date; a candidate after it is skipped. Defaults to today, so a stray future stamp never wins.

  • extensive_search (bool) – scan visible page text when no metadata carries a date, htmldate’s extensive_search. With False only the structured signals (meta, JSON-LD, time, URL) are read.

The URL helpers are successors to courlan and the w3lib.url canonicalization surface: they normalize per the WHATWG URL standard and share one frozen UrlCleaning config (how-to, courlan guide).

turbohtml.extract.clean_url(url, options=None, /)

Scrub a URL scraped from markup and normalize it, or return None when nothing usable remains.

The scrub recovers from HTML transport damage: it strips the surrounding whitespace and control characters the WHATWG basic URL parser removes (spec 4.4 steps 1-2, extended to embedded spaces since those only appear in scraped junk), unwraps <![CDATA[...]]>, truncates at a stray </>/" delimiter, and undoes a leftover &amp; escape. The survivor must be an http/https URL with a plausible host and pass the language filter, then it is returned through normalize_url().

Parameters:

  • url (str) – the URL as found in the wild.

  • options (UrlCleaning | None) – the cleaning options; defaults to UrlCleaning (drop trackers, keep slash and fragment).

Return type:

str | None

Returns:

the cleaned, normalized URL, or None for anything that is not a fetchable web URL.

Raises:

TypeError – if url is not a str.

turbohtml.extract.normalize_url(url, options=None, /)

Return the canonical form of a URL, so that two spellings of the same resource compare equal.

The spec-defined part lowercases the scheme and host, converts a Unicode host to its ASCII (punycode) form the way the URL standard’s host parser does (spec 3.5; browsers serialize münchen.de as xn--mnchen-3ya.de), drops a default port (port state), resolves ./../%2e path segments (path state), percent-encodes what the path/query/fragment percent-encode sets require – with uppercase hex digits, leaving existing escapes alone – and serializes an empty special-URL path as / (URL serializing). Beyond the spec, query parameters are sorted and known tracking parameters dropped (strict mode instead keeps only the content-identifying allowlist), and a fragment shaped like a query string is scrubbed the same way. Unlike courlan, repeated slashes are kept (the spec preserves them) and punycode is the output form, not the input form.

Parameters:

  • url (str) – an absolute or relative URL; a relative one keeps its shape, only its components are normalized.

  • options (UrlCleaning | None) – the cleaning options; defaults to UrlCleaning (drop trackers, keep slash and fragment).

Return type:

str

Returns:

the normalized URL.

Raises:

  • TypeError – if url is not a str.

  • ValueError – if the URL cannot be split into components (e.g. an unclosed IPv6 bracket) or carries a character that cannot be percent-encoded (a lone surrogate).

Collect the cleaned page links of an HTML document, the courlan.extract_links counterpart.

The document is parsed with the WHATWG tree builder, so links are read from the real DOM rather than regex matches. An anchor (<a>/<area>) contributes its href unless its rel carries nofollow or, with a language filter active, its hreflang names another language (x-default passes). Each candidate is resolved against the document base URL (a <base href> wins over base_url, per HTML spec 4.2.3), cleaned through clean_url(), and deduplicated across trivial variants (the http/https twin and the trailing-slash twin), the first occurrence in document order winning.

Parameters:

  • html (str) – the page markup.

  • base_url (str | None) – the URL the page was fetched from; relative links resolve against it, and it anchors the external/internal split. Without it relative links are dropped, since they cannot be made absolute.

  • options (UrlCleaning | None) – the cleaning options; defaults to UrlCleaning (drop trackers, keep slash and fragment).

  • external_only (bool) – keep only links leaving base_url’s site, where the site boundary is the registrable domain (the public suffix plus one label, spam.example.co.uk and example.co.uk counting as one site); the eTLD+1 is read from the shipped IANA and Public Suffix List tables, so sibling subdomains stay internal.

Return type:

set[str]

Returns:

the surviving absolute URLs.

Raises:

ValueError – if external_only is set without a base_url to define what external means.

class turbohtml.extract.UrlCleaning(strict=False, trailing_slash=True, strip_fragment=False, language=None, query_allow=None, query_deny=frozenset({}))

Options shared by clean_url(), normalize_url(), and extract_links().

Parameters:

  • strict (bool) – keep only the content-identifying query parameters (page, id, post, … plus the language parameters) instead of merely dropping known trackers, and drop the fragment. The default keeps every parameter that is not a known tracker.

  • trailing_slash (bool) – keep a path’s trailing slash. False trims it from any path but the root / when no query string follows, folding /dir/ and /dir into one form.

  • strip_fragment (bool) – always drop the fragment. The default keeps it (scrubbed of tracker parameters), since the fragment can address content (#page2, text fragments).

  • language (str | None) – an ISO 639-1 code; clean_url() and extract_links() then reject URLs whose language markers (a leading path segment such as /de/, a lang/language query parameter, or an anchor’s hreflang) point at another language. normalize_url() never rejects, so it ignores this field.

  • query_allow (frozenset[str] | None) – when set, keep only these query parameters (matched case-insensitively against the decoded name), the w3lib.url.url_query_cleaner keep-list; a listed parameter survives even when it is a known tracker. Mutually exclusive with strict, which is itself an allowlist.

  • query_deny (frozenset[str]) – always drop these query parameters (matched the same way), the url_query_cleaner remove=True mode; the tracker or strict filtering still applies to the rest.

classmethod w3lib()

Return w3lib.url.canonicalize_url’s mode: fragments dropped, every non-tracker parameter kept.

Return type:

UrlCleaning

microdata() extracts a page’s top-level HTML Microdata items, the microdata.get_items call shape (microdata guide). It returns MicrodataItem records whose get(), get_all(), and json() accessors mirror the library’s Item (documented under Structured data).

turbohtml.extract.microdata(html, base_url=None, /)

Extract a page’s top-level HTML Microdata items, the successor to microdata.get_items(html).

Each returned MicrodataItem carries the itemtype as type, the itemid as id, and the itemprop values under properties, with get(), get_all(), and json() matching the library’s Item. Shorthand for turbohtml.Document.microdata() when a page string is all you hold.

Parameters:

  • html (str) – the page markup.

  • base_url (str | None) – when given, the URL each relative URL-valued property (an a/area/link href, a media src, an object data) is resolved against, the microdata library’s base_url; a <base href> refines it. None (the default) returns every value verbatim.

Return type:

list[MicrodataItem]

Returns:

one MicrodataItem per top-level itemscope element, in document order.

Raises:

ValueError – if base_url is given and is not a valid absolute URL.

opengraph() extracts a page’s Open Graph card as an OpenGraph mapping, the opengraph library’s OpenGraph(html=...) call shape (opengraph guide).

turbohtml.extract.opengraph(html, base_url=None, /)

Extract a page’s Open Graph metadata, the successor to opengraph.OpenGraph(html=...).

Reads the og: <meta> tags turbohtml.Document.opengraph() gathers, with the twitter: tags dropped and the og: prefix stripped from each key so og["title"] reads the og:title tag. Shorthand for turbohtml.Document.opengraph() when a page string is all you hold.

Parameters:

  • html (str) – the page markup.

  • base_url (str | None) – when given, the URL each relative URL-valued property (og:url, og:image, og:video, …) is resolved against, extruct’s and metadata_parser’s absolutization; a <base href> refines it. None (the default) returns every value verbatim.

Return type:

OpenGraph

Returns:

an OpenGraph mapping of the prefix-stripped og: properties, empty when the page has none.

Raises:

ValueError – if base_url is given and is not a valid absolute URL.

class turbohtml.extract.OpenGraph(properties, /)

A page’s Open Graph metadata as a read-only mapping, the record turbohtml.Document.opengraph() returns.

Keys are the og: property names with that prefix stripped (og:title reads as og["title"]), matching the opengraph library’s OpenGraph dict. The mapping supports the full read surface – og["title"], "title" in og, og.get("title"), iteration, and equality against a plain dict – and adds is_valid().

Parameters:

properties (dict[str, str])

is_valid()

Return whether the page carries the four properties the Open Graph protocol requires, each non-empty.

Mirrors the opengraph library’s is_valid: every one of og:title, og:type, og:image, and og:url is present with a non-empty value. The opengraph_py3 fork also demands og:description; the Open Graph protocol does not, so it is not required here.

Return type:

bool