Skip to main content

Loading page…

Lexer — Glossary | LowLevelCraft

Search

Back to Glossary

Glossary Term

Lexer

A compiler component that converts raw source text into a stream of tokens (keywords, identifiers, literals, operators).

Also called a scanner or tokenizer, the lexer is the first compiler phase. Production lexers in Clang and V8 are hand-written loops applying maximal munch; generated alternatives come from tools like flex.

Related Terms

Abstract Syntax Tree Token Parser

Used In

Build 1-bit Multiplexer and Demultiplexer Token Struct Definition Basic Lexer Implementation Lexer Error Handling Identifier and Keyword Recognition The Compiler Pipeline Explained Simply How a Compiler Actually Translates Your Code