RRDtrace: Long-term raw network traffic recording using fixed-size storage

Recording raw network traffic for long-term periods can be extremely beneficial for a multitude of monitoring and security applications. However, storing all traffic of high volume networks is infeasible even for short-term periods due to the increased storage requirements. Traditional approaches for data reduction like aggregation and sampling either require knowing the traffic features of interest in advance, or reduce the traffic volume by selecting a representative set of packets uniformly over the collecting period. In this work we present RRDtrace, a technique for storing full-payload packets for arbitrary long periods using fixed-size storage. RRDtrace divides time into intervals and retains a larger number of packets for most recent intervals. As traffic ages, an aging daemon is responsible for dynamically reducing its storage space by keeping smaller representative groups of packets, adapting the sampling rate accordingly. We evaluate the accuracy of RRDtrace on inferring the flow size distribution, distribution of traffic among applications, and percentage of malicious population. Our results show that RRDtrace can accurately estimate these properties using the suitable sampling strategy, some of them for arbitrary long time and others only for a recent period.