Detect

Detect the character encoding of bytes without parsing them, a successor to chardet.detect, cchardet.detect, and charset_normalizer.from_bytes. The pipeline is the one turbohtml.parse() runs for bytes input – the WHATWG sniff (byte-order mark, then a <meta> prescan), a content detector that scores byte-pair frequencies against per-encoding character-class models, then the spec’s windows-1252 fallback – so a standalone detection and parse(data, detect_encoding=True) agree, except that a byte-order mark reports its own label here (how it decides). Decode with EncodingMatch.codec, not EncodingMatch.encoding: a WHATWG name and the CPython codec answering to it name different encodings.

turbohtml.detect.detect(data, options=None, /)[source]

Detect the character encoding of a byte string, the chardet.detect / cchardet.detect successor.

Parameters:

  • data (bytes) – the bytes to sniff; HTML input also honors a <meta> charset declaration.

  • options (Detection | None) – the detection options; defaults to Detection (always answer, no constraints).

Return type:

EncodingMatch

Returns:

the best match; its encoding is None when the input is empty or every candidate was ruled out. A leading byte-order mark reports its own label (UTF-8-SIG, UTF-16LE/BE, UTF-32LE/BE) with bom set, so a caller can strip it.

Raises:

TypeError – when data is not a bytes-like object.

turbohtml.detect.detect_all(data, options=None, /)[source]

Detect the character encoding and rank every plausible candidate, the chardet.detect_all successor.

Parameters:

  • data (bytes) – the bytes to sniff; HTML input also honors a <meta> charset declaration.

  • options (Detection | None) – the detection options; defaults to Detection (always answer, no constraints).

Return type:

list[EncodingMatch]

Returns:

the matches best first, detect()’s result leading; [EncodingMatch(None, 0.0, None)] when the input is empty or every candidate was ruled out. A leading byte-order mark collapses the ranking to one match carrying its own label and bom.

Raises:

TypeError – when data is not a bytes-like object.

class turbohtml.detect.EncodingMatch(encoding, confidence, language, bom=False, codec=None)[source]

One detection result: chardet.detect’s {"encoding", "confidence", "language"} dict as a typed record.

encoding is the WHATWG canonical name (the same string turbohtml.Document.encoding reports), or None when the input is empty or every candidate was ruled out. A leading byte-order mark reports the mark’s own label instead: "UTF-8-SIG" for a UTF-8 mark (so a caller can decode with the utf-8-sig codec to strip it), and "UTF-16LE" / "UTF-16BE" / "UTF-32LE" / "UTF-32BE" for the UTF-16 and UTF-32 marks, which a mark identifies unambiguously with no heuristic.

codec is the name to hand bytes.decode(); encoding is not. A WHATWG name and the CPython codec that answers to it are different encodings: bytes.decode("big5") reaches a strict subset of the spec’s Big5, koi8-u reaches KOI8-U where the spec means KOI8-RU, and x-mac-cyrillic reaches no codec at all. So codec names a whatwg-* codec this module registers, whose decoder is the one turbohtml.parse() uses, and data.decode(match.codec) reproduces the parser’s text. It is None with a None encoding. Those codecs decode only: encoding to a legacy charset is a separate spec algorithm turbohtml omits.

confidence is 1.0 for a certain result (a byte-order mark, a <meta> declaration, structurally valid UTF-8, escape-driven ISO-2022-JP, or pure ASCII), the candidate’s share of the positive frequency scores for a content-scored result, and 0.0 for the windows-1252 fallback chosen with no positive evidence. language names the language the winning frequency model targets ("Russian" for windows-1251, "Japanese" for Shift_JIS, …); it is None for UTF-8 and the Latin encodings whose model spans several languages. bom is true only when a byte-order mark decided the result, telling a caller the decoded text still carries the mark unless the codec strips it.

Parameters:
class turbohtml.detect.Detection(threshold=0.0, language=None, allowed=None, excluded=frozenset({}))[source]

Options for detect(), detect_all(), and EncodingDetector.

Parameters:

  • threshold (float) – the confidence floor; a candidate below it is dropped, and when every candidate falls below it the result’s encoding is None. The default 0.0 always answers, like chardet.detect.

  • language (str | None) – prefer this language when the evidence is ambiguous: candidates whose frequency model targets it (the value EncodingMatch reports as its language) rank ahead of the rest, provided they scored positively.

  • allowed (frozenset[str] | None) – when set, only these encodings (WHATWG names, any case) may be returned.

  • excluded (frozenset[str]) – these encodings are never returned; mutually exclusive with allowed.

classmethod chardet()[source]

Chardet’s UniversalDetector mode: report None under its 0.2 minimum confidence.

Return type:

Detection

class turbohtml.detect.EncodingDetector(options=None, /)[source]

Incremental detection over a byte stream, mirroring chardet’s UniversalDetector.

Call feed() with each chunk, close() for the result, and reset() to reuse the instance on another stream. done turns true as soon as the result cannot change (a leading byte-order mark, or close()), so a reader loop can stop early. Feeding buffers the chunks and close() detects once over the whole stream, so the result always equals detect() of the concatenated bytes. An instance is not thread-safe; use one per stream.

Parameters:

options (Detection | None) – the detection options; defaults to Detection (always answer, no constraints).

done: bool

Whether the result can no longer change: a leading byte-order mark was seen, or close() ran.

property result: EncodingMatch | None

The match close() computed, or None while the stream is still open.

feed(data)[source]

Buffer one chunk of the stream; ignored once done.

Parameters:

data (bytes) – the next bytes of the stream.

Return type:

None

close()[source]

Detect over everything fed so far, cache the result, and return it.

Return type:

EncodingMatch

reset()[source]

Forget the stream and the result so the instance can start over.

Return type:

None

Detect the natural language of text, a successor to whatlang, resiliparse.parse.lang, and trafilatura’s language filter. detect_language() finds the dominant Unicode script and ranks the languages sharing it by a character-trigram model, reading the visible text rather than an <html lang> attribute.

turbohtml.detect.detect_language(text, options=None, /)[source]

Detect the natural language a string is written in, the whatlang / resiliparse content-language successor.

The detector finds the dominant Unicode script, then ranks the languages that share it by the similarity between the text’s most frequent character trigrams and each language’s embedded profile. It reads the visible text only; it does not consult an <html lang> attribute.

Parameters:

  • text (str) – the text to classify; extract it from a document first (e.g. node.text).

  • options (LanguageDetection | None) – the detection options; defaults to LanguageDetection (always answer, no constraints).

Return type:

LanguageMatch

Returns:

the best match; its language is None when the text has no modeled script, every candidate was ruled out, or the confidence fell below threshold.

Raises:

TypeError – when text is not a str.

class turbohtml.detect.LanguageMatch(language, confidence, script, name=None)[source]

One language-detection result: the ISO 639-3 code, a confidence, the Unicode script, and an English name.

language is the ISO 639-3 code the text was written in ("eng", "deu", "cmn", …), or None when the text carries no script the detector models (it is empty, or all punctuation, digits, emoji, or symbols) or every candidate for its script was ruled out by LanguageDetection. confidence runs 0.0 to 1.0: it is 1.0 for a script only one modeled language uses (Greek, Georgian, Korean, …) and for a clear trigram winner, and drops toward 0.0 as the top two candidates for a shared script (Latin, Cyrillic, Arabic, Devanagari, Hebrew) converge – the usual signal that the text is too short to separate them. script is the Unicode script name the text is written in ("Latin", "Cyrillic", "Han" reported as "Mandarin", …), or None with a None language. name is the English name of the language ("English", "German", "Mandarin", …), or None with a None language.

Parameters:
class turbohtml.detect.LanguageDetection(threshold=0.0, allowed=None, excluded=frozenset({}))[source]

Options for detect_language().

Parameters:

  • threshold (float) – the confidence floor; a result below it is reported as LanguageMatch(None, 0.0, None) so a short or ambiguous input does not masquerade as a confident answer. The default 0.0 always answers.

  • allowed (frozenset[str] | None) – when set, only these languages (ISO 639-3 codes) may be returned; a text whose script has no allowed language yields no match. Constrains every script, including the single-language ones.

  • excluded (frozenset[str]) – these languages are never returned; mutually exclusive with allowed.

Normalize text to a Unicode normalization form (UAX #15), a C successor to unicodedata.normalize() and unicodedata.is_normalized(). normalize() runs all four forms – NFC, NFD, NFKC, NFKD – over the pinned tables generated from the interpreter’s own unicodedata, so the two agree exactly; a quick check returns already-normalized text untouched.

turbohtml.detect.normalize(form, text, /)[source]

Return text in a Unicode normalization form (UAX #15), the C successor to unicodedata.normalize().

The four forms pair a decomposition depth with whether the result is recomposed: NFD fully decomposes and canonically orders combining marks, NFC decomposes then recomposes, and the NFK* forms decompose compatibility equivalents too (folding ligatures, superscripts, and width variants onto their plain characters). NFC is what you want to compare or store user text so "é" written as a base plus a combining accent equals "é" written as one code point; NFKC additionally flattens presentation variants. A quick check returns already-normalized text (the common case) without allocating.

Parameters:

  • form (Literal['NFC', 'NFD', 'NFKC', 'NFKD']) – the normalization form, one of "NFC", "NFD", "NFKC", "NFKD".

  • text (str) – the string to normalize.

Return type:

str

Returns:

the normalized string, the same object when it was already normalized.

Raises:
turbohtml.detect.is_normalized(form, text, /)[source]

Return whether text is already in a Unicode normalization form, the unicodedata.is_normalized() peer.

This decides the same question as normalize(form, text) == text but the quick check settles almost every string in a single scan, without building the normalized copy.

Parameters:

  • form (Literal['NFC', 'NFD', 'NFKC', 'NFKD']) – the normalization form, one of "NFC", "NFD", "NFKC", "NFKD".

  • text (str) – the string to test.

Return type:

bool

Returns:

True when text is unchanged by normalize() for form.

Raises: