System Calls

CS 301 Lecture, Dr. Lawlor

Normally, to interact with the outside world (files, network, etc) you just call some function, usually the exact same function you'd call from C or C++.  But sometimes, such as when you're implementing a C library, or when there is no C library call to access the functionality you need, you want to talk to the OS kernel directly.  There's a special x86 "interrupt" instruction to do this, called "int". 

On Linux, you talk to the OS by loading up values into registers then calling "int 0x80".  Register rax describes what to do (open a file, write data, etc) and rbx, rcx, rdx, rsi, and rdi have the parameters describing how to do it.  This register-based parameter passing is similar to how we call functions in 64-bit x86, but the Linux kernel uses this convention both in 32 and 64 bit mode.  Other operating systems like BSD store syscall parameters on the stack, like the 32-bit x86 call interface!

Konstantin Boldyshev has a good writeup and examples of Linux, BSD, and BeOS x86 syscalls, and a list of common Linux syscalls.  (The full list of Linux syscalls is in /usr/include/asm/unistd_32.h.)  Here's a 64-bit version of his Linux example:
push rbx  ; <- we'll be using ebx below, and it's a saved register (hallelujah!)

; System calls are listed in "asm/unistd.h"
mov rax,4 ; the system call number of "write".
mov rbx,1 ; first parameter: 1, the stdout file descriptor
mov rcx,myStr ; data to write
mov rdx,3 ; bytes to write
int 0x80 ; Issue the system call

pop rbx  ; <- restore ebx to its old value 
ret

section .data
myStr:
	db "Yo",0xa

(Try this in NetRun now!)

This 64-bit version matches the way you make 32-bit Linux system calls. 

There's also a second slightly faster way to make 64-bit system calls using a *different* list of syscall numbers under /usr/include/asm/unistd_64.h.  Like the 32-bit version, the system call number is passed in rax, but the parameters are in rdi, rsi, rdx, r10, r8, r9.  Instead of "int 0x80", for this interface you use the "syscall" instruction.   The return is still in rax, and the kernel may trash rcx.

Windows system call numbers keep changing, so direct system calls aren't at all easy to use on Windows.  (This is partly a security feature, to make it harder to write portable Windows viruses.)