Longest-match tokenizer

A vocabulary maps token strings to integer ids. It always contains one special key, "UNK", whose id stands for "unknown character". "UNK" is only a sentinel: it is never matched against the text. Every other token is a non-empty string.

Implement tokenize(text: str, vocab: dict[str, int]) -> list[int], a greedy longest-match tokenizer:

• Scan the text from left to right. At the current position, find the longest vocab token (other than "UNK") that the remaining text starts with, emit its id, and move past it.
• If no token starts at the current position, emit vocab["UNK"] and move forward by one character.

vocab = {"to": 1, "token": 2, "ken": 3, "UNK": 0}
tokenize("token", vocab)     # [2]            "token" beats "to"
tokenize("tokenken", vocab)  # [2, 3]
tokenize("toxic", vocab)     # [1, 0, 0, 0]   "to", then x, i, c are unknown
tokenize("", vocab)          # []

Bound the inner loop. Trying every prefix up to the end of the text is quadratic. Compute the length of the longest real token once, and never try a longer prefix than that. The tests tokenize 200,000 characters.