Add Error Recovery: Skip Invalid Characters

Topic Overview

Recovery Keeps the Compiler Useful

When a lexer hits an illegal character, quitting leaves the user with one error and no further analysis. Clang, Rust, and TypeScript all recover: skip the bad input, emit an error token, and keep scanning so the parser can report multiple issues in one compile.

Skip and Continue

On error, record the location and message, advance past the offending character (or run of garbage), and resume normal scanning. Some lexers insert a sentinel ERROR token so the parser can synchronize at a statement boundary.

cCopy
Loading…

• Never loop forever on the same position
• Report line and column with every recovery action
• Optionally skip to next whitespace or delimiter
• Let the parser decide whether to abort or continue

Connecting to real compilers

Production compilers embed this step inside a longer pipeline. GCC flows through cpp, cc1, assembly, and ld; Clang uses the driver, Sema, LLVM IR passes, and a target backend. LLVM bitcode, JVM bytecode, and WASM are other familiar IRs at the same layer. The exercise isolates one pass so you can test it alone before chaining it to the next stage.

Why for this exercise

You will add error recovery to your lexer so invalid characters are skipped after reporting. This exercise asks you to continue producing tokens after errors, enabling multi-error diagnostics in later compiler phases.

Your Task

Make your lexer recover from bad input: instead of stopping (or flooding the user with one error per junk character), emit a single TOKEN_ERROR for each bad stretch, then carry on producing normal tokens so the parser can still see the rest of the file.

Input

A source file on stdin. Valid tokens are the ones from Basic Lexer Implementation (keywords int return if else while, the operators + - * / = == != < > <= >= && ||, delimiters ( ) { } ; ,, numbers, identifiers, // comments), plus:

• TOKEN_STRING: "..." on one line; a backslash escapes the next character.
• /* ... */ block comments, skipped like whitespace.

Recovery rules

So x@#$y is x, one error @#$, then y, and != is still a valid TOKEN_NE, while !@ is one error run.

Output

Every token, one per line, in the format of Basic Lexer Implementation:

cCopy
Loading…

Error tokens add their message:

cCopy
Loading…

Then LINE:COLUMN TOKEN_EOF and finally errors=N. Lines and columns start at 1; a newline moves to the next line, column 1; every other character is one column.

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. The lexer must always make progress: every call consumes at least one character or returns EOF.
2. Recovery is local: after an error token, the next call starts a fresh token.

Hidden tests mix all four error kinds, put junk runs directly against identifiers and operators, and leave a block comment open at EOF.