Every parser starts with a tokenizer (lexer): it turns raw text into a list of meaningful pieces so the parser never has to think about spaces or individual digits. Write one for arithmetic.
Implement tokenize(s: str) -> list:
- A run of consecutive digits is one number token, returned as an
int("120"becomes120, not1,2,0). - Each of the characters
+ - * / ( )is its own token, returned as a one-characterstr. - Whitespace (spaces, tabs, newlines) separates tokens and is otherwise skipped.
- Any other character raises
ValueError, ideally with a message naming the character and its position.
Keep it simple: a - is always the operator token "-", even before a number. Deciding whether it's unary minus is the parser's job, not the lexer's.
tokenize("12 + 3*(40-5)")
# [12, "+", 3, "*", "(", 40, "-", 5, ")"]
tokenize(" 7 ") # [7]
tokenize("-8") # ["-", 8]
tokenize("") # []
tokenize("2 ^ 3") # ValueError: unexpected character '^' at index 2
Note that "1 2" is two tokens, [1, 2]; spaces end a number.
Show hint
walk an index i along the string; when s[i] is a digit, keep advancing a second index while it's still a digit and convert the whole slice with int().