CompilersLexical AnalysisMedium

Basic Lexer Implementation

Master the fundamental concepts of lexical analysis through this focused micro-challenge.

~45 minutes Lexical Analysis Ready to learn

Topic Overview

Hand-Written Lexers in Production

Clang, GCC, V8, and Go's gc compiler all scan source with hand-rolled loops like the one you are writing. A lexer walks the input buffer, skips whitespace and comments, classifies the next lexeme, and emits a Token. Flex can generate scanners, but production compilers prefer explicit control for error messages and performance.

The Core Scan Loop

The lexer maintains a cursor into the source and repeatedly: skip trivia, peek at the current character, decide what kind of token starts here, consume characters until the lexeme is complete, then return the token with line and column metadata.

c
Loading…

Identifiers and keywords share a path: accumulate alphanumerics, then keyword-table lookup
Numbers need separate integer vs floating paths
Operators may be one or two characters (= vs ==)
EOF must be a distinct token so the parser stops cleanly

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will implement next_token() and the helper functions that drive a complete scan over a small C-like subset. The task requires a working lexer that emits the token stream your parser will consume, so focus on correct classification and position tracking.

Implementation Task

Your Task

Write the scan loop of a lexer for a small C subset. Your program reads a whole source file from stdin and prints every token it finds, one per line, followed by an end-of-file token.

Tokens

Kind	Lexemes
Keywords	`int` `return` `if` `else` `while` → `TOKEN_INT` `TOKEN_RETURN` `TOKEN_IF` `TOKEN_ELSE` `TOKEN_WHILE`
Arithmetic	`+` `-` `*` `/` `=` → `TOKEN_PLUS` `TOKEN_MINUS` `TOKEN_STAR` `TOKEN_SLASH` `TOKEN_ASSIGN`
Comparison / logic	`==` `!=` `<` `>` `<=` `>=` `&&` `\|\|` → `TOKEN_EQ` `TOKEN_NE` `TOKEN_LT` `TOKEN_GT` `TOKEN_LE` `TOKEN_GE` `TOKEN_AND` `TOKEN_OR`
Delimiters	`(` `)` `{` `}` `;` `,` → `TOKEN_LPAREN` `TOKEN_RPAREN` `TOKEN_LBRACE` `TOKEN_RBRACE` `TOKEN_SEMICOLON` `TOKEN_COMMA`
Numbers	a run of digits → `TOKEN_NUMBER`
Identifiers	`[A-Za-z_][A-Za-z0-9_]*` that is not a keyword → `TOKEN_IDENTIFIER`
Anything else	one character → `TOKEN_ERROR` (e.g. `@`, a lone `!`, a lone `&`)

Rules:

Always take the longest operator: <= is one TOKEN_LE, not < then =.
Spaces, tabs and newlines separate tokens and are never printed.
// starts a comment that runs to the end of the line.
A number stops at the first non-digit: 12ab is TOKEN_NUMBER '12' then TOKEN_IDENTIFIER 'ab' (the next task turns this into an error).

Positions

Lines and columns start at 1. A newline moves to the next line, column 1; every other character (including a tab) advances the column by 1. A token's position is the position of its first character.

Output

One line per token:

c
Loading…

then a final line with the position just past the last character:

c
Loading…

Example

Input:

c
Loading…

Output:

c
Loading…

(The input ends with a newline, so EOF sits at the start of line 4.)

Requirements

A Lexer struct holding the source, the cursor, and the current line and column.
lexer_next_token() skips whitespace and comments, then returns the next Token.
The starter's main() reads stdin and prints tokens until TOKEN_EOF; you implement the lexer.

Hidden tests cover every operator, comments at end of file, tabs, adjacent tokens with no spaces, and empty input.