Identify and Avoid Common Benchmarking Pitfalls

Topic Overview

False Wins and Silent Lies

Dead-code elimination, constant folding, comparing debug vs release builds, and measuring only one iteration are the classic ways microbenchmarks lie. LLVM and GCC will delete loops whose results are unused. Turbo Boost makes the first timed run faster than the tenth unless you pin frequency or run long enough.

Pitfalls Checklist

Always consume results. Match compiler flags across A/B tests. Warm up branch predictors and caches. Run on idle hardware or pin cores.

cCopy
Loading…

• Do not compare -O0 against -O3 and call it an optimization
• Background cron jobs add variance; check uptime and load average
• Tiny inputs fit in L1 and hide memory effects visible at production size
• Print assembly with objdump -d when numbers look impossible

Working Through the Exercise

Keep the relevant documentation open while you implement. When your output disagrees with the reference, trace one failing case by hand before changing random lines.

• Trace first: walk one failing input step by step
• Read the manual: skim the official doc for the mechanism named in this task
• Compare baselines: save measurements from the prior step so speedups are honest

Why for this exercise

You will identify at least three benchmarking mistakes in provided snippets and fix them. This exercise asks you to demonstrate one pitfall (such as DCE) and show corrected timing.

Your Task

Most wrong benchmark numbers come from a handful of mistakes. The optimiser deletes work whose result is unused, or precomputes work whose inputs are constants. The timed region is too short for the timer's resolution. Setup code is timed along with the work. The first, cold iteration is included. The machine is noisy between runs. Write a benchmark linter: given a description of a measurement, report every pitfall that applies and correct the number where possible.

Input

cCopy
Loading…

Lines may be indented, and each end prints that benchmark's report.

Checks, in this order

1. DEAD CODE if result unused.
2. CONSTANT INPUTS if inputs constant.
3. RESOLUTION if elapsed < 1000 × resolution. It reports elapsed / resolution, rounded.
4. SETUP IN TIMED REGION if setup > 0. It reports setup as a percentage of elapsed and the per-iteration time without it.
5. COLD START if first > 2 × steady. It reports first / steady.
6. TOO FAST if iterations × ops > 4 × elapsed (more than 4 operations per ns). It reports operations per ns.
7. NOISE if max > 1.1 × min. It reports (max − min) / min as a percentage.

The report starts with the measured ns per iteration (2 decimals), then one line per problem, then N problem(s) found or looks sound. All numbers are rounded half up. Missing iterations or elapsed prints need iterations and elapsed, an unrecognised key prints NAME: unknown key KEY, and a malformed line prints NAME: bad line: TEXT.

Output

cCopy
Loading…

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Evaluate each check independently, so one benchmark can show several problems.
2. Keep the numeric checks in integer arithmetic (multiply instead of dividing, for example elapsed * 4 < iterations * ops).

Hidden tests cover a clean benchmark, borderline values exactly at each threshold (which do not trigger), setup larger than the elapsed time, runs given as max then min, a benchmark missing required fields, and unknown keys.