Loading page…
The smallest meaningful unit of source code produced by lexical analysis, carrying a type, the original text, and a source location.
Tokens decouple parsing from raw characters. Precise location tracking on tokens is what enables caret diagnostics in compilers like Rust and Clang.