Skip to main navigation Skip to search Skip to main content

YSmart: Yet another SQL-to-MapReduce translator

  • Rubao Lee
  • , Tian Luo
  • , Yin Huai
  • , Fusheng Wang
  • , Yongqiang He
  • , Xiaodong Zhang
  • Ohio State University
  • Meta

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

134 Scopus citations

Abstract

MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Facebook daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called YSmart, a correlation aware SQL-to-MapReduce translator. YSmart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. YSmart can significantly reduce redundant computations, I/O operations and network transfers compared to existing translators. We have implemented YSmart with intensive evaluation for complex queries on two Amazon EC2 clusters and one Facebook production cluster. The results show that YSmart can outperform Hive and Pig, two widely used SQL-to-MapReduce translators, by more than four times for query execution.

Original languageEnglish
Title of host publicationProceedings - 31st International Conference on Distributed Computing Systems, ICDCS 2011
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages25-36
Number of pages12
ISBN (Print)9780769543642
DOIs
StatePublished - 2011

Publication series

NameProceedings - International Conference on Distributed Computing Systems
ISSN (Print)1063-6927
ISSN (Electronic)2575-8411

Fingerprint

Dive into the research topics of 'YSmart: Yet another SQL-to-MapReduce translator'. Together they form a unique fingerprint.

Cite this