Handle character encodings

Decode bytes of unknown or declared encoding the way a browser would, and inspect what the sniffer chose, with turbohtml.parse() and turbohtml.detect.detect().

Parse bytes of an unknown encoding

turbohtml.parse() accepts bytes and runs the WHATWG encoding sniffing algorithm (a byte-order mark, then a <meta> declaration, defaulting to windows-1252). Pass encoding to override the sniff, and read encoding for the WHATWG name that was resolved:

import turbohtml

doc = turbohtml.parse(b'<meta charset="iso-8859-2"><p>\xe1</p>')
print(doc.encoding)
print(doc.find("p").text)

ISO-8859-2
á

Detect an encoding without parsing

When you only need the encoding, say to decode a file or a response body, run the same sniff standalone with turbohtml.detect.detect(); it replaces chardet.detect and charset_normalizer.from_bytes:

from turbohtml.detect import detect

raw = "Précédemment, la créativité française".encode("cp1252")
match = detect(raw)
print(match.encoding, match.language)
print(raw.decode(match.codec))

windows-1252 None
Précédemment, la créativité française

Decode through match.codec, not match.encoding. The two are different strings: encoding is the WHATWG name, and the CPython codec that answers to the same name is a different encoding – bytes.decode("big5") reaches a strict subset of the spec’s Big5, koi8-u reaches KOI8-U where the spec means KOI8-RU, and x-mac-cyrillic reaches no codec at all. match.codec names a whatwg-* codec turbohtml.detect registers, whose decoder is the one turbohtml.parse() uses, so the text you get back is the text the parser would have seen.

Rank the alternatives with detect_all(), constrain or threshold them with a Detection config, and feed a stream chunk by chunk with a EncodingDetector:

from io import BytesIO

from turbohtml.detect import EncodingDetector

stream = BytesIO("\ufeffstreamed UTF-8 content".encode())
detector = EncodingDetector()
for chunk in iter(lambda: stream.read(4096), b""):
    detector.feed(chunk)
    if detector.done:  # the byte-order mark already decided the stream
        break
print(detector.close().encoding)

UTF-8-SIG

Normalize decoded text to a Unicode normalization form

Once bytes are decoded, the same visible text can carry different code points – "é" as one character or as an e plus a combining accent – so equality, search, and deduplication need a normalization pass first. turbohtml.detect.normalize() runs all four UAX #15 forms in C, the successor to unicodedata.normalize():

from turbohtml.detect import is_normalized, normalize

composed = "café"
decomposed = "café"
print(composed == decomposed)
print(normalize("NFC", decomposed) == composed)
print(is_normalized("NFC", decomposed))
print(normalize("NFKC", "file") == "file")

False
True
False
True

Use NFC to compare or store user text, NFKC to additionally flatten presentation variants (ligatures, superscripts, width variants), and the NFD / NFKD forms when you want the fully decomposed representation. is_normalized() answers the membership question without building the normalized copy.