Distributed processing of very large datasets with DataCutter

A DataCutter framework that is designed to provide support for subsetting and processing of datasets in a distributed and heterogeneous environment is presented. The use of DataCutter with several data-intensive applications from diverse fields was illustrated. The experimental results demonstrate the impact of heterogeneity on an application, and further suggest that any static application organization will likely not perform efficiently in all cases. The DataCutter filtering service uses techniques such as careful placement of filters, multiple filter group instances, and transparent copies to adjust dynamically to the heterogeneity present in the targeted runtime environment.