Reverse Engineer a Simple Custom File Format

Topic Overview

Reverse Engineering Unknown File Formats

When you encounter an undocumented binary file, you must deduce its structure without specifications. This skill applies to malware analysis (custom protocols), game modding (proprietary assets), and data recovery (legacy formats).

Deduction Techniques

Magic bytes at offset 0 identify the type. Even custom formats often have a version identifier there.

Structure fields reveal themselves through patterns:

• A 4-byte value matching file size minus header suggests a length field
• Two values that look like width and height suggest image dimensions
• Offsets into the file are usually smaller than total file size

For example, if bytes at offset 8 show 34 12 and the expected value is 0x1234, the file is little-endian.

Hex editing workflow: dump the first 256 bytes, identify magic, guess field sizes, validate by checking that offsets point to plausible data.

For This Exercise

You will document techniques for identifying magic bytes, structure fields, and endianness in unknown formats. This is the same process Wireshark dissectors and game mod tools use when no specification exists.

Validating Field Guesses

After guessing field layout, check that offsets point to plausible data: strings should be printable, lengths should not exceed file size, and checksums should validate if present. Change one field at a time in a hex editor and observe whether the target program still parses the file. This iterative hypothesis-testing approach is how modding communities reverse game asset formats without official documentation.

Write a C program that parses an unknown file supplied as a hex dump, applying the magic-bytes-then-fields-then-offsets methodology.

Header layout (little-endian throughout):

• Bytes 0-3: magic "CUST" (hex 43 55 53 54)
• Byte 4: version
• Bytes 5-6: data_size, u16 little-endian
• Bytes 7-8: data_offset, u16 little-endian
• Payload: data_size ASCII bytes starting at data_offset

Input:

• One line: the byte count N, whitespace, then the file as hex pairs (upper or lower case)

Output on a valid file, exactly: magic: CUST (ok) version: <v> data_size: <s> data_offset: <o> payload: <payload characters> payload_sum: <sum of payload byte values, decimal>

Failure cases (reject paths):

• Fewer than 9 bytes total: print "header: truncated" then "reject"
• Magic is not "CUST": print "magic: invalid" then "reject"
• data_offset + data_size exceeds the byte count: print "payload: out of bounds" then "reject"

Success Criteria:

• Code compiles with gcc -Wall -Wextra
• Little-endian decoding, bounds checking, and the byte-sum are all computed, not hardcoded