TY - GEN
T1 - A data locality aware online scheduling approach for I/O-intensive jobs with file sharing
AU - Khanna, Gaurav
AU - Catalyurek, Umit
AU - Kure, Tahsin
AU - Sadayappan, P.
AU - Saltz, Joel
PY - 2007
Y1 - 2007
N2 - Many scientific investigations have to deal with large amounts of data from simulations and experiments. Data analysis in such investigations typically involves extraction of subsets of data, followed by computations performed on extracted data. Scheduling in this context requires efficient utilization of the computational, storage and network resources to optimize response time. The data-intensive nature of such applications necessitates data-locality aware job scheduling algorithms. This paper proposes a hypergraph based dynamic scheduling heuristic for a stream of independent I/O intensive jobs with file sharing behavior. The proposed heuristic is based on an event-driven, run-time hypergraph modeling of the file sharing characteristics among jobs. Our experiments on a coupled compute/storage cluster show it performs better compared to previously proposed strategies, under a varying set of parameters for workloads from the application domain of biomedical image analysis.
AB - Many scientific investigations have to deal with large amounts of data from simulations and experiments. Data analysis in such investigations typically involves extraction of subsets of data, followed by computations performed on extracted data. Scheduling in this context requires efficient utilization of the computational, storage and network resources to optimize response time. The data-intensive nature of such applications necessitates data-locality aware job scheduling algorithms. This paper proposes a hypergraph based dynamic scheduling heuristic for a stream of independent I/O intensive jobs with file sharing behavior. The proposed heuristic is based on an event-driven, run-time hypergraph modeling of the file sharing characteristics among jobs. Our experiments on a coupled compute/storage cluster show it performs better compared to previously proposed strategies, under a varying set of parameters for workloads from the application domain of biomedical image analysis.
UR - https://www.scopus.com/pages/publications/38049100526
U2 - 10.1007/978-3-540-71035-6_7
DO - 10.1007/978-3-540-71035-6_7
M3 - Conference contribution
AN - SCOPUS:38049100526
SN - 9783540710349
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 141
EP - 160
BT - Job Scheduling Strategies for Parallel Processing - 12th International Workshop, JSSPP 2006, Revised Selected Papers
PB - Springer Verlag
T2 - 12th Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2006
Y2 - 26 June 2006 through 26 June 2006
ER -