Network-level polymorphic shellcode detection using emulation

Significant progress has been made in recent years towards preventing code injection attacks at the network level. However, as state-of-the-art attack detection technology becomes more prevalent, attackers are likely to evolve, employing techniques such as polymorphism and metamorphism to defeat these defenses. A major outstanding question in security research and engineering is thus whether we can proactively develop the tools needed to contain advanced polymorphic and metamorphic attacks. While recent results have been promising, most of the existing proposals can be defeated using only minor enhancements to the attack vector. In fact, some publicly-available polymorphic shellcode engines are currently one step ahead of the most advanced publicly-documented network-level detectors. In this paper, we present a heuristic detection method that scans network traffic streams for the presence of previously unknown polymorphic shellcode. In contrast to previous work, our approach relies on a NIDS- embedded CPU emulator that executes every potential instruction sequence in the inspected traffic, aiming to identify the execution behavior of polymorphic shellcode. Our analysis demonstrates that the proposed approach is more robust to obfuscation techniques like self-modifications compared to previous proposals, but also highlights advanced evasion techniques that need to be more closely examined towards a satisfactory solution to the polymorphic shellcode detection problem.