TY - GEN
T1 - Autoscaling for hadoop clusters
AU - Gandhi, Anshul
AU - Thota, Sidhartha
AU - Dube, Parijat
AU - Kochut, Andrzej
AU - Zhang, Li
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2016/6/1
Y1 - 2016/6/1
N2 - Unforeseen events such as node failures and resource contention can have a severe impact on the performance of data processing frameworks, such as Hadoop, especially in cloud environments where such incidents are common. SLA compliance in the presence of such events requires the ability to quickly and dynamically resize infrastructure resources. Unfortunately, the distributed and stateful nature of data processing frameworks makes it challenging to accurately scale the system at run-time. In this paper, we present the design and implementation of a model-driven autoscaling solution for Hadoop clusters. We first develop novel gray-box performance models for Hadoop workloads that specifically relate job execution times to resource allocation and workload parameters. We then employ these models to dynamically determine the resources required to successfully complete the Hadoop jobs as per the user-specified SLA under various scenarios including node failures and multi-job executions. Our experimental results on three different Hadoop cloud clusters and across different workloads demonstrate the efficacy of our models and highlight their autoscaling capabilities.
AB - Unforeseen events such as node failures and resource contention can have a severe impact on the performance of data processing frameworks, such as Hadoop, especially in cloud environments where such incidents are common. SLA compliance in the presence of such events requires the ability to quickly and dynamically resize infrastructure resources. Unfortunately, the distributed and stateful nature of data processing frameworks makes it challenging to accurately scale the system at run-time. In this paper, we present the design and implementation of a model-driven autoscaling solution for Hadoop clusters. We first develop novel gray-box performance models for Hadoop workloads that specifically relate job execution times to resource allocation and workload parameters. We then employ these models to dynamically determine the resources required to successfully complete the Hadoop jobs as per the user-specified SLA under various scenarios including node failures and multi-job executions. Our experimental results on three different Hadoop cloud clusters and across different workloads demonstrate the efficacy of our models and highlight their autoscaling capabilities.
KW - AutoScaling
KW - Hadoop
KW - Performance Modeling
UR - https://www.scopus.com/pages/publications/84978079713
U2 - 10.1109/IC2E.2016.11
DO - 10.1109/IC2E.2016.11
M3 - Conference contribution
AN - SCOPUS:84978079713
T3 - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016
SP - 109
EP - 118
BT - Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 4th IEEE Annual International Conference on Cloud Engineering, IC2E 2016
Y2 - 4 April 2016 through 8 April 2016
ER -