Write Microbenchmark Framework with Warmup and Statistics

Topic Overview

Warmup, Iterations, and Statistics

Google Benchmark, LLVM's llvm-exegesis, and Julia's @benchmark all share the same skeleton: warmup, many timed iterations, then aggregate statistics. Without warmup, first-touch page faults and cold icache skew the first samples. Without variance reporting, a lucky 5% speedup from turbo noise looks like a win.

Framework Structure

Separate setup from the timed body. Loop enough iterations that timer overhead is below 1% of measured time. Track min, median, and standard deviation.

cCopy
Loading…

• Compile with -O2 or -O3 unless you explicitly test debug behavior
• Prevent dead-code elimination with volatile or blackhole sinks
• Reset state between iterations when the work mutates inputs
• Log environment: CPU model, governor, ASLR on/off

Working Through the Exercise

Keep the relevant documentation open while you implement. When your output disagrees with the reference, trace one failing case by hand before changing random lines.

• Trace first: walk one failing input step by step
• Read the manual: skim the official doc for the mechanism named in this task
• Compare baselines: save measurements from the prior step so speedups are honest

Why for this exercise

You will build a microbenchmark harness with configurable warmup and iteration counts plus mean/min output. This exercise requires honest statistics so you can tell signal from measurement noise.

Your Task

The statistics are the heart of a microbenchmark framework (Google Benchmark, Criterion, JMH). You discard warmup iterations (cold caches, frequency ramp-up), report robust statistics (median and p90, not just the mean), remove outliers caused by interrupts, flag unstable runs by their coefficient of variation, and refuse to call a difference real when the ranges overlap. Implement that engine over recorded per-iteration timings.

Input

cCopy
Loading…

Rules

• Drop the first K samples, then sort the rest.
• median: the middle value, or the average of the two middle values (rounded half up).
• p90, Q1, Q3: nearest rank, i.e. the value at 1-based rank ceil(P * n / 100).
• mean: rounded half up. sd: the sample standard deviation, the integer square root (rounded down) of sum((x − mean)^2) / (n − 1) (0 for one sample).
• Outliers: values outside [Q1 − 3·IQR/2, Q3 + 3·IQR/2] (integer division), where IQR = Q3 − Q1. Recompute mean and sd without them.
• CV = sd / mean of the kept samples, 1 decimal, rounded half up. Add (unstable: CV above 5%) if CV > 5.0%.
• compare A B: B's median / A's median (2 decimals), and whether the kept ranges [min, max] of the two overlap.
• Errors: warmup: 0..100, analyze NAME: no samples, not enough samples after warmup, and compare: analyze both benchmarks first.

Output

cCopy
Loading…

The overlap case reads (ranges overlap: the difference may be noise). The warmup count shown is capped at the number of samples.

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Compute everything on a sorted copy that excludes the warmup samples. Never on the raw order.
2. Keep all statistics in integer arithmetic, so the results are reproducible.

Hidden tests cover an even sample count (median of two), a noisy run flagged as unstable, overlapping ranges, a warmup that swallows every sample, a single sample, and comparing before analysing.