User-Level Infrastructure for System Call Interposition: A Platform for Intrusion Detection and Confinement

Several new approaches for detecting malicious attacks on computer systems and/or confining untrusted or malicious applications have emerged over the past several years. These techniques often rely on the fact that when a system is attacked from a remote location over a network, damage can ultimately be inflicted only via system calls made by processes running on the target system. This factor has lead to a surge of interest in developing infrastructures that enable secure interception and modification of system calls made by processes running on the target system. Most known approaches for solving this problem have relied on an in-kernel approach, where the interception mechanisms as well as the intrusion detection/confinement systems are implemented within the operating system kernel. We explore an alternative approach that uses mechanisms provided by most variants of the UNIX operating system to implement system call interposition at user level, where the system calls made by one process are monitored by another process. Some of the key problems that need to solved in developing such an approach are: providing adequate set of capabilities in the infrastructure, portability of the security enhancements and the infrastructure itself across different operating systems, and minimizing performance overheads associated with interception for a wide range of applications. We present a solution that satisfactorily addresses these issues, and can thus lead to a platform for rapid development and deployment of robust intrusion detectors, confinement systems and other application-specific security enhancements.