Loading page…
Master the fundamental concepts of lexical analysis through this focused micro-challenge.
You have read the whole brief, and the concepts above stay free on every task. Writing and running the code needs a plan.
Three hints are available for this task, revealed one at a time inside the code workspace so you can struggle productively before seeing them.
Every task includes starter code, theory, and hidden tests so you can implement and verify locally in the browser.
How it worksIdentifiers and keywords look identical in the source: a letter followed by letters and digits. The lexer reads the full spelling, then checks a keyword table. Clang and GCC both use hash tables or trie structures for this lookup; CPython caches interned identifier strings.
Start at the first alphabetic character (or underscore). Consume while characters are alphanumeric or underscore. Buffer the spelling. If the spelling matches a reserved word in the keyword table, emit a keyword token; otherwise emit an identifier token carrying the name.
cLoading…
While and while differentlyProduction compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.
You will implement identifier scanning and keyword disambiguation in your lexer. The exercise asks you to read alphanumeric sequences, consult a keyword table, and emit the correct token type so the parser can distinguish int the keyword from int as a variable name.
Scan a source file and report every word, each keyword and identifier, with its position. This isolates the one lexer path where most subtle bugs live: telling int from int123, _int or INT.
A C source file on stdin.
A word starts with a letter or _ and continues through letters, digits and _. It is a keyword if it exactly matches (case-sensitive) one of these 13:
cLoading…
Otherwise it is an identifier. Everything else is skipped and prints nothing, with three rules so you never report a word that isn't really one:
_, so 9lives and 0x1f yield no word.// ... to end of line and /* ... */ are skipped entirely."..." and '...' are skipped entirely; a backslash escapes the next character. (Inputs never leave these unterminated.)Lines and columns start at 1; a newline moves to the next line, column 1; every other character is one column.
One line per word, in order:
cLoading…
where TOKEN_KIND is TOKEN_ + the keyword in upper case (TOKEN_INT, TOKEN_SIZEOF, …) or TOKEN_IDENTIFIER. Finish with:
cLoading…
Input:
cLoading…
Output:
cLoading…
scan_identifier() consumes one word and returns its length.lookup_keyword() maps a word to its keyword kind, or TOKEN_IDENTIFIER.Hidden tests cover case sensitivity (INT, Return), keywords as prefixes (int123, forward, constant), numbers glued to letters, and words inside comments, strings and char literals.