Loading page…
Master the fundamental concepts of lexical analysis through this focused micro-challenge.
Every production compiler starts by carving source text into tokens. Clang's Token class, GCC's cpp_token, and CPython's tokenizer all carry the same core fields: a kind, the original lexeme, and a source location. Get this struct wrong and every later phase pays for it.
A token is the smallest meaningful unit the parser consumes. For int x = 42; the lexer must emit distinct tokens for the keyword, identifier, operator, literal, and semicolon. Each token needs enough metadata to drive parsing and diagnostics.
cLoading…
KW_INT with lexeme intIDENT with lexeme xINT_LIT with value 42SEMI with lexeme ;Line and column numbers are what let Rust and Clang print caret diagnostics instead of vague file-level errors.
Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.
You will implement the Token and TokenType definitions that every subsequent lexer function returns. This exercise asks you to nail down the data contract before writing scanning logic: enum variants for keywords, identifiers, literals, and punctuation, plus a struct wide enough to hold literal values and source positions.
Define the TokenType enum and Token struct for a small C lexer, plus the helpers that create, classify, print and free tokens. There is no scanning yet: a "scanner" has already split the source into lexemes and hands you one per line.
Zero or more lines, each of the form:
cLoading…
LINE and COLUMN are integers. LEXEME is everything after the second space: it never needs further splitting, but a string literal may contain spaces (4 16 "done here").
| Lexeme | TokenType |
|---|---|
int return if else while | TOKEN_INT TOKEN_RETURN TOKEN_IF TOKEN_ELSE TOKEN_WHILE |
+ - * / = | TOKEN_PLUS TOKEN_MINUS TOKEN_STAR TOKEN_SLASH TOKEN_ASSIGN |
== != < > <= >= | TOKEN_EQ TOKEN_NE TOKEN_LT TOKEN_GT TOKEN_LE TOKEN_GE |
( ) { } ; , | TOKEN_LPAREN TOKEN_RPAREN TOKEN_LBRACE TOKEN_RBRACE TOKEN_SEMICOLON TOKEN_COMMA |
digits only (42, 007) | TOKEN_NUMBER: value is the decimal integer |
starts and ends with ", length ≥ 2 | TOKEN_STRING: value is the text between the quotes |
letter or _, then letters/digits/_ | TOKEN_IDENTIFIER |
anything else (12abc, @, "oops) | TOKEN_ERROR |
Keywords win over identifiers: while is TOKEN_WHILE, whilex is an identifier. The enum must also contain TOKEN_EOF.
One line per input token:
cLoading…
where TYPE is the enumerator name exactly as written above. Numbers and strings append their union value: value=42 or value=done here (no quotes). After the last token print a summary, even when the input is empty:
cLoading…
M counts TOKEN_ERROR tokens.
Input:
cLoading…
Output:
cLoading…
Token holds the type, an owned copy of the lexeme, line, column, and a union with long int_val (numbers) and char *str_val (strings).token_create() allocates the token, copies the lexeme, and fills the union from the lexeme.token_destroy() frees everything token_create() allocated (including str_val).token_type_to_string() returns the enumerator name, e.g. "TOKEN_GE".main() is already written in the starter: you only fill in the types and helpers.Hidden tests use the same format with more token kinds, malformed lexemes, and empty input.
Three hints are available for this task, revealed one at a time inside the code workspace so you can struggle productively before seeing them.
Every task includes starter code, theory, and hidden tests so you can implement and verify locally in the browser.
How it works