The Computer Journal Advance Access originally published online on September 2, 2008
The Computer Journal 2009 52(4):429-460; doi:10.1093/comjnl/bxn043
| ||||||||||||||||||||||||||||||||||||||||||||||||
A Pragmatic Methodology for Testing Intrusion Prevention Systems
1 Yahoo! Inc., Santa Clara, CA 95054, USA
2 University of Athens, Athens, 15784, Greece
3 Fortinet Inc., Sunnyvale, CA 94086, USA
* Corresponding author: ad{at}di.uoa.gr
Received 20 July 2007; revised 21 May 2008
Intrusion prevention systems (IPSs) not only attempt to detect attacks but also block malicious traffic and pro-actively tear down pertinent network connections. To effectively thwart attacks, IPSs have to operate both in real-time and inline fashion. This dual mode renders the design/implementation and more importantly the testing of IPSs a challenge. In this paper, we propose an IPS testing framework termed IPS Evaluator which consists of a trace-driven inline simulator-engine, mechanisms for generating and manipulating test cases, and a comprehensive series of test procedures. The engine features attacker and victim interfaces which bind to the external and internal ports of an IPS-under-testing (IUT). Our engine employs a bi-directional injection policy to ensure that replayed packets are subject to security inspection by the IUT before they are forwarded. Furthermore, the send-and-receive mechanism of our engine allows for the correlation of engine-replayed and IUT-forwarded packets as well as the verification of IUT actions on detected attacks. Using dynamic addressing and routing techniques, our framework rewrites both source and destination addresses for every replayed packet on-the-fly. In this way, replayed packets conform to the specific features of the IUT. We propose algorithms to partition attacker/victim-emanated packets so that they are subjected to security inspections by the IUT and in addition, we offer packet manipulation operations to shape replayed traces. We discuss procedures that help verify the IUT's detection and prevention accuracy, attack coverage and behavior under diverse traffic patterns. Finally, we evaluate the strengths of our framework by mainly examining the open-source IPS Snort-Inline. IPS deficiencies revealed during testing help establish the effectiveness of our approach.
Key Words: testing of intrusion prevention systems • testing methodology • inline operation • detection and prevention accuracy of IPSs