Understand Instruction Encoding: Manually Encode 3 x86 Instructions

Topic Overview

Manual x86 Encoding

x86 instructions are byte sequences built from prefixes, opcode, ModR/M, SIB, displacement, and immediate fields. Disassemblers walk these bytes; encoders emit them. Understanding the format is prerequisite to writing tools like yours.

cCopy
Loading…

ModR/M Byte

The ModR/M byte names two registers and an addressing mode:

• Mod: memory vs register-direct
• Reg/Opcode: register operand or opcode extension
• R/M: second register or base for memory

For example, 89 C3 is MOV EBX, EAX in 32-bit mode: opcode 89, ModR/M places source and destination registers.

For this exercise, you will hand-encode three instructions and verify with objdump -d or the mini disassembler. This task asks you to treat bytes as structured data, the same skill malware analysts use when carving shellcode.

Working Through the Exercise

Keep the relevant datasheet, ISA manual, or architecture textbook chapter open while you implement. When your output disagrees with the reference trace on the same program, the bug is usually a mis-decoded opcode, a stale register read, or a flag bit left unchanged after arithmetic.

• Trace first: walk one failing input by hand before changing random lines
• Read the manual: skim the official doc for the mechanism named in this task
• Compare baselines: save measurements from the prior step so speedups are honest

For this exercise, you will use those habits while implementing the requirement in the starter code. Microarchitectural product names change across CPU generations, but the control ideas (fetch, bypass, cache lines, vector lanes) stay stable enough to debug from first principles.

Your Task

Write a small x86-64 assembler for a handful of instructions and show how every byte is built: the REX prefix, the opcode, the ModR/M byte and the immediate. After this you'll be able to read objdump -d byte columns, and you'll see why x86 instructions are anywhere from 1 to 15 bytes long.

Registers

A 64-bit operation gets the prefix REX.W = 48. Both operands of a two-register instruction must be the same size.

Encodings

ModR/M is mod(2 bits) reg(3) rm(3). Immediates are little-endian two's complement. Immediates are decimal, possibly negative, in the signed 32-bit range.

Output

One line per instruction, bytes in lower-case hex:

cCopy
Loading…

The bracket lists, in order, REX.W (if present), op= (as b8+r/50+r/58+r for the register-in-opcode forms), modrm=mod.reg.rm in binary (if present) and imm8=0xHH / imm32=0xHHHHHHHH (if present). The instruction is echoed as mnemonic op1, op2, normalised to single spaces. A line that can't be encoded prints LINE => error. After the last line print total: N bytes.

Example

Input:

cCopy
Loading…

Output:

cCopy
Loading…

Requirements

1. Build each instruction into a byte buffer, then print it; keep ModR/M construction in one helper.
2. Choose the shortest encoding exactly as the table says (imm8 before the eax short form before 81).

Hidden tests cover negative imm8, the 81 form for a non-accumulator register, mov with immediates in both sizes, inc/dec/pop/nop, and lines that can't be encoded (mixed sizes, push of a 32-bit register, unknown mnemonics).