Clean¶
Clean untrusted or raw HTML: sanitize it against an allowlist and rewrite bare URLs into links. Sanitizing is a
successor to bleach.clean – build a Policy (or take a preset), then sanitize; a non-overridable baseline
removes scripting elements, event-handler attributes, and javascript: URLs regardless of the policy. Linkifying is a
successor to bleach.linkify – it finds URLs and email addresses and wraps them
in <a> links, HTML-aware so it never links inside an existing <a>, a raw-text element, or a caller’s
skip_tags.
- turbohtml.clean.sanitize(html, options=None)¶
Sanitize an HTML fragment against a policy.
- Parameters:
- Return type:
- Returns:
the sanitized, safe HTML.
- Raises:
TypeError – if a set-typed policy field (
tags,url_schemes,remove_with_content,css_properties,attribute_prefixes,media_hosts) holds a value that is not a set or frozenset, orattribute_prefixescontains a non-string.ValueError – if
attribute_prefixescontains an empty string, which would match every attribute.
- class turbohtml.clean.Sanitizer(options=None)¶
A reusable sanitizer; build it once from a
Policyand callsanitize()from any thread.- sanitize(html)¶
Sanitize an HTML fragment.
- Parameters:
html (
str) – the untrusted HTML fragment.- Return type:
- Returns:
the sanitized, safe HTML.
- Raises:
TypeError – if a set-typed policy field (
tags,url_schemes,remove_with_content,css_properties,attribute_prefixes,media_hosts) holds a value that is not a set or frozenset, orattribute_prefixescontains a non-string.ValueError – if
attribute_prefixescontains an empty string, which would match every attribute.
- class turbohtml.clean.Policy(tags=frozenset({'a', 'abbr', 'acronym', 'b', 'blockquote', 'code', 'em', 'i', 'li', 'ol', 'strong', 'ul'}), attributes=<factory>, url_schemes=frozenset({'http', 'https', 'mailto'}), allow_relative_urls=True, on_disallowed_tag=OnDisallowed.ESCAPE, strip_comments=True, add_link_rel=frozenset({}), attribute_filter=None, set_attributes=<factory>, remove_with_content=frozenset({}), css_properties=frozenset({'azimuth', 'background-color', 'border-bottom-color', 'border-collapse', 'border-color', 'border-left-color', 'border-right-color', 'border-top-color', 'clear', 'color', 'cursor', 'direction', 'display', 'elevation', 'fill', 'fill-opacity', 'fill-rule', 'float', 'font', 'font-family', 'font-size', 'font-style', 'font-variant', 'font-weight', 'height', 'letter-spacing', 'line-height', 'overflow', 'pause', 'pause-after', 'pause-before', 'pitch', 'pitch-range', 'richness', 'speak', 'speak-header', 'speak-numeral', 'speak-punctuation', 'speech-rate', 'stress', 'stroke', 'stroke-linecap', 'stroke-linejoin', 'stroke-opacity', 'stroke-width', 'text-align', 'text-decoration', 'text-indent', 'unicode-bidi', 'vertical-align', 'voice-family', 'volume', 'white-space', 'width'}), attribute_prefixes=frozenset({}), attribute_values=<factory>, media_hosts=frozenset({}))¶
An immutable, thread-safe description of what sanitizing keeps.
Build one and reuse it across threads. Whatever a policy allows, a non-configurable baseline still removes the unsafe element set, event-handler attributes, and dangerous URL schemes, so every policy is safe by construction.
- Parameters:
tags (
frozenset[str]) – the allowed element set; any tag outside it is handled peron_disallowed_tag.attributes (
Mapping[str,frozenset[str]]) – allowed attribute names keyed by tag ("*"as the key matches every tag, and"*"inside a set allows every name).url_schemes (
frozenset[str]) – the allowlist for URL-bearing attributes such ashrefandsrc.allow_relative_urls (
bool) – keep relative (scheme-less) URLs, which carry no scheme to check.on_disallowed_tag (
OnDisallowed) – how to treat a tag not intags(OnDisallowed: escape, strip, or remove).strip_comments (
bool) – drop HTML comments from the output.add_link_rel (
frozenset[str]) –reltokens forced onto every kept<a href>(e.g.noopener).attribute_filter (
Callable[[str,str,str],str|None] |None) – an optional last word over every surviving attribute, returning a replacement value orNoneto drop it.set_attributes (
Mapping[str,Mapping[str,str]]) – attribute values forced onto every kept instance of a tag (added if absent, overwritten if present); unlikeattribute_filter, this can add attributes that were not there.remove_with_content (
frozenset[str]) – disallowed tags whose whole subtree is dropped (e.g.script/style) rather than escaped or stripped, so their text never leaks into the output.css_properties (
frozenset[str]) – the CSS property allowlist. A keptstyleattribute and, whenstyleis intags, the<style>element’s stylesheet body are both scrubbed against it: any declaration whose property name is not in the set (or whose value smugglesexpression()or aurl()with a disallowed scheme) is dropped, while selectors and block nesting are kept, so dangerous CSS cannot ride in on a keptstyle.attribute_prefixes (
frozenset[str]) – allow any attribute whose name starts with one of these prefixes (e.g."data-"for everydata-*), on top of the exact-name and"*"matches inattributes.attribute_values (
Mapping[str,Mapping[str,frozenset[str]]]) – restrict a kept attribute to literal values, keyed{tag: {attribute: allowed_values}}; a surviving attribute whose value is outside its set is dropped. This narrows an attributeattributesalready admits and cannot admit a new one.media_hosts (
frozenset[str]) – allowed hosts for an embedded-mediasrc(audio,video,source,track); asrcwhose URL host is not one of these lowercase entries is dropped. Empty means no host restriction.
- classmethod strict()¶
Allow no markup at all: every tag is escaped to text and every attribute dropped.
- Return type:
- Returns:
the strict policy.
- classmethod basic()¶
Allow bleach’s default 12-tag set, for migration parity.
- Return type:
- Returns:
the basic policy.
- class turbohtml.clean.OnDisallowed(*values)¶
What to do with an element the policy does not allow.
ESCAPErenders the tag as visible text (<x>becomes<x>), the safe bleach default that keeps the content readable;STRIPdrops the tag but keeps its children;REMOVEdrops the tag and its whole subtree.
The sanitizer ships bleach’s default allowlists as module constants, so a Policy can extend a known baseline
instead of enumerating a safe set from scratch.
- turbohtml.clean.DEFAULT_TAGS¶
Build an immutable unordered collection of unique elements.
The tags the default policy keeps:
a,abbr,acronym,b,blockquote,code,em,i,li,ol,strong,ul.
- turbohtml.clean.DEFAULT_ATTRIBUTES¶
Read-only proxy of a mapping.
The attributes the default policy keeps, keyed by tag:
hrefandtitleona, andtitleonabbrandacronym.
- turbohtml.clean.DEFAULT_SCHEMES¶
Build an immutable unordered collection of unique elements.
The URL schemes the default policy allows in an
hreforsrc:http,https,mailto.
- turbohtml.clean.DEFAULT_CSS_PROPERTIES¶
Build an immutable unordered collection of unique elements.
The CSS properties the default policy keeps when scrubbing a
styleattribute: the CSS 2.1 safe set plus the SVG paint properties.
Linkifying¶
A Linkify configuration object carries the knobs: a callback receives each generated LinkCandidate and
returns it to keep the link or None to leave the text bare, process_existing runs the callbacks over <a>
tags already in the input (a callback reads LinkCandidate.existing to tell the two apart), extra_tlds extends
bare-domain detection beyond the built-in IANA table, and schemes sets which explicit-scheme URLs autolink
(defaulting to the built-in http/https/ftp set, so a typo scheme or a javascript:// payload stays plain
text).
- turbohtml.clean.linkify(text, options=None)¶
Find URLs and email addresses in HTML and wrap them in
<a>links, leaving existing markup untouched.
- class turbohtml.clean.Linkify(callbacks=(<function nofollow>, ), skip_tags=None, parse_email=False, process_existing=False, extra_tlds=None, schemes=None)¶
An immutable, thread-safe description of how
linkify()andLinkerfind and rewrite links.Build one and reuse it across threads.
- Parameters:
callbacks (
Iterable[Callable[[LinkCandidate],LinkCandidate|None]]) – callables run on each detected link to adjust or veto it (defaults toDEFAULT_CALLBACKS).skip_tags (
Iterable[str] |None) – tags whose text is left untouched, such aspreandcode.parse_email (
bool) – also autolink bare email addresses asmailto:links.process_existing (
bool) – run the callbacks over<a>tags already present, not only freshly detected links.extra_tlds (
Iterable[str] |None) – top-level domains that make a bare domain a link, on top of the built-in IANA table.schemes (
Iterable[str] |None) – the exact set ofscheme://URL schemes that autolink;Nonekeeps the built-inhttp/https/ftpdefault, so a typo scheme or ajavascript://payload stays plain text. A bare domain is always treated ashttpand is governed by the TLD table, notschemes.
- class turbohtml.clean.Linker(options=None)¶
A reusable linkifier; build it once from a
Linkifyconfiguration and calllinkify()per document.
- class turbohtml.clean.LinkCandidate(url, text, attrs=None, *, existing=False)¶
A link handed to each callback to mutate or veto.
A callback returns the link to keep it, or
Noneto drop the anchor: a detected link stays plain text, an existing one is unwrapped to its contents.
- turbohtml.clean.nofollow(link)¶
Add
rel="nofollow"to a web link so search engines skip it, leavingmailto:and other links alone.- Parameters:
link (
LinkCandidate) – the link to adjust.- Return type:
- Returns:
the link, with
nofollowadded when it is a web link.
- turbohtml.clean.target_blank(link)¶
Open a web link in a new tab, stripping a stale
targetfrom a non-web link so it cannot leak through.- Parameters:
link (
LinkCandidate) – the link to adjust.- Return type:
- Returns:
the link, with
targetset on a web link or cleared on a non-web link.
- turbohtml.clean.Callback¶
str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str
Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to ‘utf-8’. errors defaults to ‘strict’.
The type of a linkify callback: a callable that takes one
LinkCandidateand returns it to keep the link orNoneto leave the text bare.nofollow()andtarget_blank()are built-in examples.
- turbohtml.clean.DEFAULT_CALLBACKS¶
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple. If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
The callbacks
linkify()applies when the caller passes none:(nofollow,), so bare-URL links getrel="nofollow"unless you opt out.
To only locate links in plain text rather than rewrite HTML, use LinkDetector. It returns a LinkSpan
for each match and accepts custom tlds and scheme-less schemes.
- class turbohtml.clean.LinkDetector(*, emails=True, bare_domains=True, tlds=(), schemes=())¶
Find the links in plain text, configured once and reused per call.
Unlike
Linker, which rewrites HTML, a detector only locates links and hands backLinkSpanobjects, leaving the text untouched.- Parameters:
emails (
bool) – detect bare email addresses.bare_domains (
bool) – detect bare domains (example.com) with no explicit scheme.tlds (
Iterable[str]) – custom top-level domains accepted for bare-domain matching, on top of the IANA table.schemes (
Iterable[str]) – extra schemes to detect, both as scheme-less opaque URLs (tel:,bitcoin:) and asscheme://authority URLs, on top of the built-inhttp/https/ftpset; an unregistered scheme such asjavascript://or a typo likehppt://is not detected.
- find(text)¶
Find every link in a run of text.
- class turbohtml.clean.LinkSpan(start, end, text, url, is_email)¶
One URL or email address found in a run of plain text.
- Parameters:
start (
int) – the half-open start offset of the match in the scanned text.end (
int) – the half-open end offset of the match in the scanned text.text (
str) – the matched substring exactly as it appeared.url (
str) – the normalizedhref(mailto:for an email,http://for a bare domain, the text itself for ascheme://or registered scheme-less URL).is_email (
bool) – whether the match is an email address.
Minifying¶
minify() shrinks an HTML document in one call – it parses the input and serializes it through the round-trip-safe
Minify layout, so the output reparses to the same tree and minifying is idempotent
(minify(minify(x)) == minify(x)). It replaces minify-html and htmlmin. The four transforms (fold
insignificant whitespace, omit optional tags, unquote attributes, strip comments) default on; pass a
Minify to turn any off.
- turbohtml.clean.minify(html, options=None)¶
Minify an HTML document.
Parse
htmlinto a tree and serialize it with the minifying layout, folding insignificant whitespace, omitting the start/end tags the WHATWG rules make optional, dropping redundant attribute quotes, and stripping comments. Each transform is round-trip safe, so the result reparses to the same tree and minifying is idempotent.
The HTML minify layout emits <style> bodies verbatim; to also minify embedded CSS, run minify_css() (below)
over a <style> body yourself, which is what minify-html’s minify_css did inline. Passing a JSMinify
as Minify(minify_js=...) rewrites inline <script> content, covering minify-html’s minify_js. The doctype
is always normalized to <!doctype html> (minify-html’s minify_doctype is implicit), and HTML has no
processing instructions to drop (remove_processing_instructions is moot under the WHATWG parser, which reads them as
bogus comments).
JS minification¶
minify_js() minifies a JavaScript string on its own, the jsmin/rjsmin successor. It always folds
whitespace, comments, and number literals; a frozen JSMinify toggles the optional mangle (rename local
bindings) and fold (constant-fold and eliminate dead code) passes. The same JSMinify passed as
Minify(minify_js=...) rewrites inline <script> content during HTML minification.
- turbohtml.clean.minify_js(source, options=None, on_error='raise')¶
Minify a JavaScript source string, returning the shortest equivalent program.
Every transform preserves semantics, so the result evaluates identically to
source.optionsdefaults to the full pipeline (fold and mangle).on_errorchooses what happens when the parser cannot handle the script. The default"raise"fails loudly;"passthrough"returnssourceunchanged instead, the never-fail contractjsminandrjsminguarantee – the same verbatim fallback the inline-<script>path already applies so one bad script cannot break serialization.- Parameters:
- Return type:
- Returns:
the minified JavaScript, or
sourceverbatim when it is unparsable andon_error="passthrough".- Raises:
ValueError – if
on_erroris not"raise"or"passthrough", or – whenon_error="raise"– if the parser cannot handle the script (a lexical or syntax error, or input nested past the depth limit), with the message naming the construct, its byte offset, and the offending token.
- class turbohtml.clean.JSMinify(mangle=True, fold=True)¶
Which optional JavaScript-minifier passes to run.
Whitespace and numeric-literal minification is unconditional, as is comment removal – except
/*! ... */bang comments and any comment carrying an@licenseor@preserveannotation, which are kept byte-exact as a leading banner so a license header survives (the same rule the CSS minifier applies).manglerenames local bindings to short names (the bulk of the size win) andfoldruns constant folding and dead-code elimination; turning either off keeps that aspect of the source readable (e.g.mangle=Falsefor debuggable output).
CSS minification¶
Minify CSS the value-safe way: every transform produces output that parses to the same cascade. minify_css() takes
a whole stylesheet; minify_css_inline() takes a bare declaration list, the value of an HTML style attribute.
Both are value-safe at any baseline; the optional CSSMinify baseline year only bounds how new the output
syntax may be.
- turbohtml.clean.minify_css(css, options=None)¶
Minify a full CSS stylesheet.
- turbohtml.clean.minify_css_inline(css, options=None)¶
Minify an inline declaration list, the value of an HTML
styleattribute.Use this rather than
minify_css()when the source is bare declarations (color:red; margin:0) with no surrounding selector or braces.
- class turbohtml.clean.CSSMinify(baseline=None)¶
Options for
minify_css()andminify_css_inline().baselineis the Baseline year the output may target: the minifier applies a transform whose output syntax reached Baseline in yearYonly whenbaseline >= Y.None(the default) emits only long-interoperable syntax;2021additionally merges the shorthands that reached Baseline that year (inset, the flexgap, the two-valueoverflow). Every year is value-safe – the year bounds only how new the output syntax may be, never the cascade it parses to.
Transformations¶
Every transformation below preserves the computed value: the output parses to the same cascade as the input on any conformant browser. Each links to the specification that establishes the equivalence.
Numbers and dimensions¶
Drop a leading
+, redundant leading and trailing zeros, and switch toe-notation when it is shorter (Syntax 3 §4.3.3, Values 4 §6.1).a{width:+0.50px}→a{width:.5px},a{margin:100000px}→a{margin:1e5px}.Lower-case a known unit and drop the unit on a zero
<length>(0px→0); angle, time, frequency and other dimensions keep their unit, since a bare0is a<length>only (Values 4 §5.2).a{margin:0PX}→a{margin:0}, whilea{transform:rotate(0deg)}is unchanged.Fold a
calc()of constant, like-united operands with exact rational arithmetic; a non-combinable or non-terminating result is kept verbatim, and+/-without surrounding whitespace is left untouched (Values 4 §10).a{width:calc(1px + 2px)}→a{width:3px}.
Colors¶
Swap a named color and its hex for whichever is shorter, and shorten
#rrggbb→#rgband#rrggbbaa→#rgba(Color 4 §5.2, §6.1).a{color:#ffffff}→a{color:#fff},a{color:#000080}→a{color:navy}.Fold an opaque
rgb()/hsl()to hex only when every channel lands on an exact 8-bit value; a fractional channel is kept functional, since rounding it changes the color (Color 4 §15).a{color:rgb(255,0,0)}→a{color:red}.Collapse
transparentandrgba(0,0,0,0)to#0000, drop an alpha of1, and use the shorterrgb()/hsl()alias ofrgba()/hsla()(Color 4 §4).a{color:transparent}→a{color:#0000},a{color:rgba(1,2,3,.5)}→a{color:rgb(1,2,3,.5)}.
Shorthands¶
Collapse a 1–4 value box shorthand when mirrored edges are equal, and merge the four physical longhands back into the shorthand (Box 3 §6–7, Cascade 5 §2.2).
a{margin:1px 1px 1px 1px}→a{margin:1px}.Collapse the
background,background-position,background-repeat(Backgrounds 3 §3),flex(Flexbox 1 §7.1.1) andfont(Fonts 4 §2.7) shorthands to their shortest equivalent form.a{font:bold 12px x}→a{font:700 12px x}.Merge
flex-direction+flex-wrapintoflex-flow, and each Box Alignment axis pair into itsplace-shorthand –align-content+justify-contentintoplace-content,align-items+justify-itemsintoplace-items, andalign-self+justify-selfintoplace-self(Box Alignment 3).a{align-items:center;justify-items:center}→a{place-items:center}.Merge the four
border-*-radiuscorners intoborder-radius(Backgrounds 3 §6) andoutline-width+outline-style+outline-colorintooutline(UI 4 §3.1); each resets only its own longhands, and the corner merge stands down for an elliptical corner or a logical sibling.a{outline-width:1px; outline-style:solid;outline-color:red}→a{outline:1px solid red}.
Structure and selectors¶
Collapse insignificant whitespace and strip comments, keeping a
/*! … */bang comment (Syntax 3 §3.3)./*c*/a{color:red}→a{color:red}, while/*! keep */a{color:red}→/*!keep*/a{color:red}.Drop a declaration an identically-keyed later one overrides, remove an empty rule or an empty
@media/@supports/@container(an empty@layeror@keyframesis kept, since either is observable), merge two rules with the same selector or an identical body – even across intervening rules, as long as each sets none of the moved properties so the cascade cannot change – and fuse consecutive@mediablocks that share a prelude (Cascade 5 §6.4.4).a{}b{c:d}→b{c:d},@media print{}is dropped, anda{color:red}b{margin:0}a{font-size:2px}→a{color:red;font-size:2px}b{margin:0}.Lower-case type selectors, trim combinator whitespace, drop a redundant universal
*before a subclass, write the four legacy pseudo-elements with one colon (::before→:before), and unquote an attribute value that is a valid identifier (Selectors 4 §5–6, Pseudo-Elements 4 §8); a custom-property name keeps its case (Variables 1 §2).Keep a custom-property value byte-exact past edge trimming: §2 defines the value as the literal token stream,
var()splices it verbatim, andgetComputedStyle().getPropertyValue()reads it back as written, so collapsing its internal whitespace – which the other PyPI minifiers do for a few hundred bytes on custom-property-heavy CSS – is observable and never applied.A{X:1}→a{x:1},[a="Foo"]{x:1}→[a=Foo]{x:1}.Rewrite a
@keyframesfromselector to0%(Animations 1), and drop the space beforeand/orafter a)in a media query (Media Queries 4).@keyframes k{from{opacity:0}}→@keyframes k{0%{opacity:0}}.
Baseline¶
Every transform above is value-safe and, by default, applies to every stylesheet – its output syntax has been
interoperable for years. A few emit a shorthand whose interop is more recent, so they are gated on the
CSSMinify baseline year and stay off unless you target that year or later. Each row lists the year a
transform’s output syntax reached Baseline; a transform tagged year Y runs when
baseline >= Y.
Baseline year |
Transforms enabled |
|---|---|
any ( |
Every transform in the sections above: numbers, colors, box/flex/ |
|
Merge |
turbohtml.migration.bleach¶
A drop-in for bleach.clean for projects migrating off bleach. It translates bleach’s arguments onto a
Policy; the safety baseline still applies, so an attributes callable cannot re-admit an
event handler or a javascript: URL.
- turbohtml.migration.bleach.clean(text, tags=None, attributes=None, protocols=None, strip=False, strip_comments=True, css_sanitizer=None)¶
Sanitize HTML against a bleach-style allowlist, the way
bleach.cleandid.- Parameters:
text (
str) – the untrusted HTML.tags (
Iterable[str] |None) – the allowed tag names; None uses bleach’s default set.attributes (
object) – the allowed attributes as a flat list, a per-tag dict, or a (tag, name, value) predicate; None uses bleach’s default.protocols (
Iterable[str] |None) – the allowed URL schemes; None uses bleach’s default.strip (
bool) – drop a disallowed tag and keep its children, rather than escaping it to text.strip_comments (
bool) – drop HTML comments from the output.css_sanitizer (
object) – accepted for signature compatibility; passing one raisesNotImplementedError.
- Return type:
- Returns:
the sanitized, safe HTML.