Hermes: An Optimization of HyperLogLog Counting in real-time data processing

Yunxiang Zhao; Songtao Guo; Yuanyuan Yang

doi:10.1109/IJCNN.2016.7727430

Hermes: An Optimization of HyperLogLog Counting in real-time data processing

Yunxiang Zhao
, Songtao Guo
, Yuanyuan Yang

Electrical Engineering

National University of Defense Technology
Southwest University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

HyperLogLog Counting is widely used in cardinality estimation. It is the foundation of many algorithms in data analysis, commodity recommendation and database optimization. Facing the large scale internet business like electronic commerce, internet companies have an urgent requirement of distributed real-time cardinality estimation with high accuracy and low time cost. In this paper, we propose a distributed real-time cardinality estimation algorithm named Hermes. Hermes adjusts the estimated cardinality dynamically according to the result of HyperLogLog Counting and also optimizes the data distribution strategy of existing distributed cardinality estimation algorithms. Experiments have been carried out and the results show that Hermes has lower estimation error and time cost compared with existing algorithms.

Original language	English
Title of host publication	2016 International Joint Conference on Neural Networks, IJCNN 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1890-1895
Number of pages	6
ISBN (Electronic)	9781509006199
DOIs	https://doi.org/10.1109/IJCNN.2016.7727430
State	Published - Oct 31 2016
Event	2016 International Joint Conference on Neural Networks, IJCNN 2016 - Vancouver, Canada Duration: Jul 24 2016 → Jul 29 2016

Publication series

Name	Proceedings of the International Joint Conference on Neural Networks
Volume	2016-October

Conference

Conference	2016 International Joint Conference on Neural Networks, IJCNN 2016
Country/Territory	Canada
City	Vancouver
Period	07/24/16 → 07/29/16

Access to Document

10.1109/IJCNN.2016.7727430

Cite this

Zhao, Y., Guo, S., & Yang, Y. (2016). Hermes: An Optimization of HyperLogLog Counting in real-time data processing. In 2016 International Joint Conference on Neural Networks, IJCNN 2016 (pp. 1890-1895). Article 7727430 (Proceedings of the International Joint Conference on Neural Networks; Vol. 2016-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IJCNN.2016.7727430

@inproceedings{4877f995954b4587ba72b32ecc73e8f7,

title = "Hermes: An Optimization of HyperLogLog Counting in real-time data processing",

abstract = "HyperLogLog Counting is widely used in cardinality estimation. It is the foundation of many algorithms in data analysis, commodity recommendation and database optimization. Facing the large scale internet business like electronic commerce, internet companies have an urgent requirement of distributed real-time cardinality estimation with high accuracy and low time cost. In this paper, we propose a distributed real-time cardinality estimation algorithm named Hermes. Hermes adjusts the estimated cardinality dynamically according to the result of HyperLogLog Counting and also optimizes the data distribution strategy of existing distributed cardinality estimation algorithms. Experiments have been carried out and the results show that Hermes has lower estimation error and time cost compared with existing algorithms.",

author = "Yunxiang Zhao and Songtao Guo and Yuanyuan Yang",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 International Joint Conference on Neural Networks, IJCNN 2016 ; Conference date: 24-07-2016 Through 29-07-2016",

year = "2016",

month = oct,

day = "31",

doi = "10.1109/IJCNN.2016.7727430",

language = "English",

series = "Proceedings of the International Joint Conference on Neural Networks",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1890--1895",

booktitle = "2016 International Joint Conference on Neural Networks, IJCNN 2016",

}

Zhao, Y, Guo, S & Yang, Y 2016, Hermes: An Optimization of HyperLogLog Counting in real-time data processing. in 2016 International Joint Conference on Neural Networks, IJCNN 2016., 7727430, Proceedings of the International Joint Conference on Neural Networks, vol. 2016-October, Institute of Electrical and Electronics Engineers Inc., pp. 1890-1895, 2016 International Joint Conference on Neural Networks, IJCNN 2016, Vancouver, Canada, 07/24/16. https://doi.org/10.1109/IJCNN.2016.7727430

Hermes: An Optimization of HyperLogLog Counting in real-time data processing. / Zhao, Yunxiang; Guo, Songtao; Yang, Yuanyuan.
2016 International Joint Conference on Neural Networks, IJCNN 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 1890-1895 7727430 (Proceedings of the International Joint Conference on Neural Networks; Vol. 2016-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Hermes

T2 - 2016 International Joint Conference on Neural Networks, IJCNN 2016

AU - Zhao, Yunxiang

AU - Guo, Songtao

AU - Yang, Yuanyuan

PY - 2016/10/31

Y1 - 2016/10/31

N2 - HyperLogLog Counting is widely used in cardinality estimation. It is the foundation of many algorithms in data analysis, commodity recommendation and database optimization. Facing the large scale internet business like electronic commerce, internet companies have an urgent requirement of distributed real-time cardinality estimation with high accuracy and low time cost. In this paper, we propose a distributed real-time cardinality estimation algorithm named Hermes. Hermes adjusts the estimated cardinality dynamically according to the result of HyperLogLog Counting and also optimizes the data distribution strategy of existing distributed cardinality estimation algorithms. Experiments have been carried out and the results show that Hermes has lower estimation error and time cost compared with existing algorithms.

AB - HyperLogLog Counting is widely used in cardinality estimation. It is the foundation of many algorithms in data analysis, commodity recommendation and database optimization. Facing the large scale internet business like electronic commerce, internet companies have an urgent requirement of distributed real-time cardinality estimation with high accuracy and low time cost. In this paper, we propose a distributed real-time cardinality estimation algorithm named Hermes. Hermes adjusts the estimated cardinality dynamically according to the result of HyperLogLog Counting and also optimizes the data distribution strategy of existing distributed cardinality estimation algorithms. Experiments have been carried out and the results show that Hermes has lower estimation error and time cost compared with existing algorithms.

UR - https://www.scopus.com/pages/publications/85007195976

U2 - 10.1109/IJCNN.2016.7727430

DO - 10.1109/IJCNN.2016.7727430

M3 - Conference contribution

AN - SCOPUS:85007195976

T3 - Proceedings of the International Joint Conference on Neural Networks

SP - 1890

EP - 1895

BT - 2016 International Joint Conference on Neural Networks, IJCNN 2016

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 24 July 2016 through 29 July 2016

ER -

Hermes: An Optimization of HyperLogLog Counting in real-time data processing

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this