Understand Syscall ABI: Registers, Numbers, Kernel Entry

Topic Overview

Crossing the User-Kernel Boundary

The syscall ABI defines how CPU state crosses privilege rings. On x86-64 Linux, user code loads rax with the syscall number, places arguments in designated registers, executes syscall, and reads the return from rax.

Register Map

x86-64 Linux passes syscall arguments in fixed registers:

• Syscall number: rax
• Arg 1: rdi, Arg 2: rsi, Arg 3: rdx
• Arg 4: r10, Arg 5: r8, Arg 6: r9

For example, read(3, buf, 1024) maps to NR 0, fd in rdi, buffer in rsi, count in rdx.

Why This Matters in Production

The r10-instead-of-rcx quirk you'll learn here exists because the x86-64 syscall instruction itself clobbers rcx and r11 to hold the return address and flags, a detail glibc's syscall wrappers and Go's runtime assembly both have to account for by hand. This ABI is exactly what objdump -d reveals when reverse-engineering a stripped binary, and mismatching it is a classic bug in hand-written syscall stubs for new libc ports.

Debugging Checklist

Before you call the implementation done, walk failure modes on purpose. Test empty structures, single-element edge cases, maximum concurrency, and errno paths that must not crash the program. OS code usually fails in production when happy-path tests pass but invariants break under contention or memory pressure.

Implementation Notes

Keep structures small and name fields after kernel counterparts when possible. That lets you read man pages and kernel source side by side while you work. Print observable events during development; remove noisy logs once tests pass reliably.

For This Exercise

You will document the ABI for three syscalls and write a tiny assembly stub that invokes one. This exercise requires explaining why rcx and r11 are clobbered by the syscall instruction itself.

Your Task

Every architecture has its own system-call ABI: which register holds the syscall number, which registers hold the arguments, which instruction traps into the kernel, and which numbers name which calls. Build a translator for three ABIs. It decodes a register snapshot into a readable call, encodes a call into register assignments, and compares syscall numbers across ABIs.

Input

cCopy
Loading…

Rules

• Pointer arguments (buf, pathname, addr) print in hex. Everything else prints as a signed integer at the ABI's register width, so on i386 0xffffff9c decodes to -100.
• Decode warnings, printed before the result: warning: R is not a syscall register on ABI. The special case rcx on x86_64 prints warning: 4th argument goes in r10, not rcx (syscall overwrites rcx). With no number register the result is ABI: no syscall number in REG, and an unknown number gives ABI: unknown syscall number N.
• encode i386 mmap becomes mmap2, whose last argument counts 4096-byte pages. An offset that is not a multiple of 4096 is an error. Encoding a call the ABI lacks prints encode ABI CALL: no such syscall on ABI, with (use openat) added for open. A wrong argument count prints expected N argument(s), got K.
• compare shows i386 mmap as i386=192 (mmap2).

Output

cCopy
Loading…

Encode lines are " %-3s = %-18s ; %s", and the instruction line is " %-24s ; result in %s".

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Make each ABI a data record (registers, instruction, number table), with one decoder and one encoder shared by all three.
2. Interpret non-pointer values at the ABI's register width. A 32-bit ABI must sign-extend from bit 31.

Hidden tests cover six-argument mmap on all three ABIs, the mmap2 page conversion and its error, i386 sign extension, the rcx mistake on x86_64, registers from the wrong ABI, missing number registers, calls missing on arm64, wrong argument counts, and unknown ABIs.