A program or subroutine that performs lexical analysis, converting a sequence of characters into a sequence of tokens.
"The lexer, or lexical analyzer, breaks down the source code into meaningful symbols that the parser can understand."
— @openai · February 25, 2024