A performance prediction framework for data intensive applications on large scale parallel machines

This paper presents a simulation-based performance predic- tion framework for large scale data-intensive applications on large scale machines. Our framework consists of two components: application emula- tors and a suite of simulators. Application emulators provide a paramete- rized model of data access and computation patterns of the applications and enable changing of critical application components (input data parti- tioning, data declustering, processing structure, etc.) easily and flexibly. Our suite of simulators model the I/O and communication subsystems with good accuracy and execute quickly on a high-performance work- station to allow performance prediction of large scale parallel machine configurations. The key to eficient simulation of very large scale confi- gurations is a technique called loosely-coupled simulation where the pro- cessing structure of the application is embedded in the simulator, while preserving data dependencies and data distributions. We evaluate our performance prediction tool using a set of three data-intensive applica- tions.