Tokenizing a document

Go from a string of HTML to a stream of tokens you can inspect.

Start with a small document and hand it to turbohtml.tokenize(), which returns an iterator of turbohtml.Token objects:

import turbohtml

for token in turbohtml.tokenize('<p class="intro">Tom &amp; Jerry</p>'):
    print(token)

Token(START_TAG, tag='p')
Token(TEXT, data='Tom & Jerry')
Token(END_TAG, tag='p')

type identifies each token as a turbohtml.TokenType. Start and end tags carry the lowercased tag name and the attributes, decoded:

start, text, end = turbohtml.tokenize('<p class="intro">Tom &amp; Jerry</p>')
print(start.type)
print(start.tag)
print(start.attrs)

1
p
[('class', 'intro')]

Text arrives with character references resolved (the &amp; above came through as a plain &). Content that the HTML specification treats as raw, such as a script body, arrives as one text token without further interpretation:

print([
    token.data
    for token in turbohtml.tokenize("<script>if (a < b) run()</script>")
    if token.type is turbohtml.TokenType.TEXT
])

['if (a < b) run()']

When the document arrives in pieces (from a network stream, for example), create a turbohtml.Tokenizer and feed the pieces as they come. Each feed() returns the tokens that piece completed, and close() flushes whatever remains:

tokenizer = turbohtml.Tokenizer()
print([token.tag for token in tokenizer.feed("<div><sp")])
print([token.tag for token in tokenizer.feed("an>")])
print(list(tokenizer.close()))

['div']
['span']
[]

The incomplete <sp stayed buffered until the rest of the tag arrived. That is the whole tokenizer API. If you are porting an existing html.parser.HTMLParser subclass, turbohtml.migration.stdlib.HTMLParser keeps the same handle_* callbacks over this tokenizer, so the migration is changing the base class. Head to the How-to guides guides for task-focused recipes or the Reference for the exact signatures.