Implement Custom System Call in Linux Kernel

Topic Overview

Extending the Kernel Interface

Custom syscalls start in kernel space: assign a syscall number, implement SYSCALL_DEFINE handler, and expose via the syscall table. Userspace needs a wrapper or raw syscall() with the new NR.

Kernel Side Checklist

Steps on x86_64:

• Add prototype in syscalls.h / arch table
• Implement handler validating user pointers with copy_from_user
• Register number unused in your kernel version docs

For example, a sys_hello returning a constant might use NR 467 on a teaching kernel while mainline Linux reserves official numbers centrally.

Why This Matters in Production

The SYSCALL_DEFINEx macro and syscall table registration you're documenting here is the real onboarding path the Linux kernel requires for every new syscall (io_uring and pidfd_open both went through this exact process), and it's also why kernel CVEs so often trace back to a missing copy_from_user bounds check on a new syscall's arguments. Once assigned, a syscall number is effectively permanent ABI , removing io_uring's early numbers would break every binary that already calls them.

Debugging Checklist

Before you call the implementation done, walk failure modes on purpose. Test empty structures, single-element edge cases, maximum concurrency, and errno paths that must not crash the program. OS code usually fails in production when happy-path tests pass but invariants break under contention or memory pressure.

Implementation Notes

Keep structures small and name fields after kernel counterparts when possible. That lets you read man pages and kernel source side by side while you work. Print observable events during development; remove noisy logs once tests pass reliably.

For This Exercise

You will add a minimal syscall returning data to userspace and call it from a test program. This exercise requires discussing why unvalidated pointers from user mode are security bugs.

Your Task

Adding a system call to Linux takes two pieces of machinery. The first is the syscall table, generated from arch/x86/entry/syscalls/syscall_64.tbl. The second is the SYSCALL_DEFINEn macro, which wraps your function so the entry code can call it with a struct pt_regs. Rebuild both: parse table lines, generate the dispatch table for an ABI, dispatch syscall numbers, and expand SYSCALL_DEFINEn into the three functions it creates.

Input

cCopy
Loading…

Rules

• Parsing. NR must be 0-1023, and ABI must be one of the three. Otherwise print line N: number X out of range, line N: unknown abi X or line N: cannot parse (N counts every input line).
• build ABI uses lines whose ABI is common or equal to ABI. The table has max NR + 1 slots. A slot with an ENTRY gets __x64_ENTRY (__x32_ENTRY for x32 lines). Holes and reserved lines get sys_ni_syscall. If two selected lines share a number, print build ABI: duplicate number NR (NAME1 and NAME2); after that, call has no table. Runs of consecutive sys_ni_syscall slots print as ranges.
• call compares NR as unsigned with the table size, so negative numbers are out of range.
• define allows at most 6 arguments, because only 6 argument registers exist (di si dx r10 r8 r9). __se_sys_ takes every argument as long (this is how the kernel sign-extends them), and __do_sys_ casts them back. SYSCALL_DEFINE0 creates only the __x64_sys_ function.

Output

cCopy
Loading…

In the __do_sys_ prototype, a type ending in * is followed directly by the name, and any other type is followed by a space. Other messages: build X: unknown abi, define: expected NAME followed by TYPE, ARG pairs, define NAME: at most 6 arguments (only 6 argument registers).

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Fill every slot the table does not name with sys_ni_syscall, so dispatch is a plain bounds check followed by an array index.
2. Map argument i of the macro to register i of di, si, dx, r10, r8, r9 (r10, not rcx, because syscall clobbers rcx).

Hidden tests cover the x32 table (with __x32_ entries and x32-only duplicates), reserved numbers, unknown ABIs and unparsable lines, call before any build and after a failed build, a six-argument define, a seven-argument define, malformed defines, and large or negative numbers.