#######
Clean
#######
.. module:: turbohtml.clean
Clean untrusted or raw HTML: sanitize it against an allowlist and rewrite bare URLs into links. Sanitizing is a
successor to ``bleach.clean`` -- build a :class:`Policy` (or take a preset), then sanitize; a non-overridable baseline
removes scripting elements, event-handler attributes, and ``javascript:`` URLs regardless of the policy. Linkifying is a
successor to `bleach.linkify `_ -- it finds URLs and email addresses and wraps them
in ```` links, HTML-aware so it never links inside an existing ````, a raw-text element, or a caller's
``skip_tags``.
.. autofunction:: sanitize
:func:`sanitize_report` sanitizes and also returns what the policy dropped, one :class:`Removed` record per removed
element or stripped attribute, the way DOMPurify populates ``DOMPurify.removed``.
.. autofunction:: sanitize_report
.. autoclass:: Removed
.. autoclass:: Sanitizer
:members: sanitize, sanitize_report
.. autoclass:: Policy
:members: strict, basic, relaxed
``Policy.css_properties`` allowlists style property *names*; ``Policy.allowed_styles`` narrows further by *value*, the
way sanitize-html's ``allowedStyles`` does. Key it ``{tag: {property: [pattern, ...]}}`` (``"*"`` matches every tag); a
``style`` declaration survives only when its property is listed for the element's tag or ``"*"`` and its value matches
one of the patterns via an unanchored :func:`re.search`. It runs on top of ``css_properties`` and the dangerous-value
baseline -- the property must still be in ``css_properties``, and ``expression()`` or a disallowed-scheme ``url()`` is
dropped even when a pattern would admit it:
.. testcode::
from turbohtml.clean import sanitize, Policy
policy = Policy(
tags=frozenset({"p"}),
attributes={"p": frozenset({"style"})},
css_properties=frozenset({"color"}),
allowed_styles={"*": {"color": [r"^#[0-9a-f]{3,6}$"]}},
)
print(sanitize('ok
', policy))
.. testoutput::
ok
``Policy.transform_tags`` renames elements during the same walk, sanitize-html's ``transformTags``. Key it by source
tag: map to a bare string to rename, or to a :class:`Transform` to rename and add attributes. The rename runs *before*
the allowlist, so the renamed element is re-checked from scratch -- a transform decides an element's name but never its
safety. Mapping a tag to ``script`` still drops it, and an added attribute is scrubbed like the element's own, so it
must be allowlisted to survive:
.. testcode::
from turbohtml.clean import sanitize, Policy, Transform
policy = Policy(
tags=frozenset({"strong", "div"}),
attributes={"div": frozenset({"class"})},
transform_tags={"b": "strong", "center": Transform("div", {"class": "center"})},
)
print(sanitize("bold and middle", policy))
.. testoutput::
bold and middle
``Policy.isolate_named_props`` prefixes every kept ``id`` and ``name`` value with ``user-content-``, DOMPurify's
``SANITIZE_NAMED_PROPS``. It stops DOM clobbering -- an ``id`` or ``name`` whose value matches a built-in ``document``
or form property shadows that property through named access -- by moving the value out of the property namespace. An
already-prefixed value is left alone, so re-sanitizing is a fixpoint. The isolation is applied after
``attribute_filter``, so the value a filter returns is the one that gets namespaced:
.. testcode::
from turbohtml.clean import sanitize, Policy
policy = Policy(
tags=frozenset({"input"}),
attributes={"input": frozenset({"name"})},
isolate_named_props=True,
)
print(sanitize('', policy))
.. testoutput::
``Policy.custom_element_check`` keeps an unlisted hyphenated custom element when its matcher admits the tag name,
DOMPurify's ``CUSTOM_ELEMENT_HANDLING.tagNameCheck``; ``custom_attribute_check`` does the same for that element's
attributes (``attributeNameCheck``), and ``allow_customized_builtins`` keeps an ``is`` attribute naming a custom
element. The safety baseline still runs on whatever the matcher keeps, so an ``on*`` handler is dropped even here:
.. testcode::
from turbohtml.clean import sanitize, Policy
policy = Policy(
tags=frozenset({"p"}),
custom_element_check=lambda tag: tag.startswith("x-"),
custom_attribute_check=lambda _tag, name: name.startswith("data-"),
)
print(sanitize('c
', policy))
.. testoutput::
c
``Policy.allow_html``, ``Policy.allow_svg``, and ``Policy.allow_mathml`` gate each namespace independently, DOMPurify's
``USE_PROFILES``. All default on; turning one off drops that whole namespace even when its tags are allowlisted, so a
policy can keep SVG but not MathML:
.. testcode::
policy = Policy(tags=frozenset({"svg", "circle", "math", "mi"}), allow_svg=False)
print(sanitize("", policy))
.. testoutput::
<svg><circle></circle></svg>
``Policy.xml`` serializes the cleaned tree as well-formed XML/XHTML rather than HTML. The walk is unchanged, so the
policy is exactly as safe; the difference is output syntax -- an empty element self-closes, values follow the XML
escaping rules, a foreign root declares its namespace, and any comment, control character, or attribute name XML cannot
hold is neutralized -- so the result reparses through :func:`turbohtml.parse_xml`. Use it to feed an XHTML dialect that
rejects HTML's bare ``
``:
.. testcode::
policy = Policy(tags=frozenset({"p", "br"}), xml=True)
print(sanitize("one
two
", policy))
.. testoutput::
one
two
.. autoclass:: Transform
.. autoclass:: OnDisallowed
:members:
The sanitizer ships bleach's default allowlists as module constants, so a :class:`Policy` can extend a known baseline
instead of enumerating a safe set from scratch.
.. autodata:: DEFAULT_TAGS
:no-value:
The tags the default policy keeps: ``a``, ``abbr``, ``acronym``, ``b``, ``blockquote``, ``code``, ``em``, ``i``,
``li``, ``ol``, ``strong``, ``ul``.
.. autodata:: DEFAULT_ATTRIBUTES
:no-value:
The attributes the default policy keeps, keyed by tag: ``href`` and ``title`` on ``a``, and ``title`` on ``abbr``
and ``acronym``.
.. autodata:: DEFAULT_SCHEMES
:no-value:
The URL schemes the default policy allows in an ``href`` or ``src``: ``http``, ``https``, ``mailto``.
.. autodata:: DEFAULT_CSS_PROPERTIES
:no-value:
The CSS properties the default policy keeps when scrubbing a ``style`` attribute: the CSS 2.1 safe set plus the SVG
paint properties.
************
Linkifying
************
A :class:`Linkify` configuration object carries the knobs: a callback receives each generated :class:`LinkCandidate` and
returns it to keep the link or ``None`` to leave the text bare, ``process_existing`` runs the callbacks over ````
tags already in the input (a callback reads ``LinkCandidate.existing`` to tell the two apart), ``extra_tlds`` extends
bare-domain detection beyond the built-in IANA table, and ``schemes`` sets which explicit-scheme URLs autolink
(defaulting to the built-in ``http``/``https``/``ftp`` set, so a typo scheme or a ``javascript://`` payload stays plain
text).
.. autofunction:: linkify
.. autoclass:: Linkify
:members:
.. autoclass:: Linker
:members: linkify
.. autoclass:: LinkCandidate
.. autofunction:: nofollow
.. autofunction:: target_blank
.. autodata:: Callback
:no-value:
The type of a linkify callback: a callable that takes one :class:`LinkCandidate` and returns it to keep the link or
``None`` to leave the text bare. :func:`nofollow` and :func:`target_blank` are built-in examples.
.. autodata:: DEFAULT_CALLBACKS
:no-value:
The callbacks :func:`linkify` applies when the caller passes none: ``(nofollow,)``, so bare-URL links get
``rel="nofollow"`` unless you opt out.
To only *locate* links in plain text rather than rewrite HTML, use :class:`LinkDetector`. It returns a :class:`LinkSpan`
for each match and accepts custom ``tlds`` and scheme-less ``schemes``.
.. autoclass:: LinkDetector
:members: find, has_link
.. autoclass:: LinkSpan
:members:
***********
Minifying
***********
:func:`minify` shrinks an HTML document in one call -- it parses the input and serializes it through the round-trip-safe
:class:`~turbohtml.Minify` layout, so the output reparses to the same tree and minifying is idempotent
(``minify(minify(x)) == minify(x)``). It replaces ``minify-html`` and ``htmlmin``. The four transforms (fold
insignificant whitespace, omit optional tags, unquote attributes, strip comments) default on; pass a
:class:`~turbohtml.Minify` to turn any off.
.. autofunction:: minify
The HTML minify layout emits ``