Skip to main navigation Skip to search Skip to main content

The IceProd framework: Distributed data processing for the IceCube neutrino observatory

  • M. G. Aartsen
  • , R. Abbasi
  • , M. Ackermann
  • , J. Adams
  • , J. A. Aguilar
  • , M. Ahlers
  • , D. Altmann
  • , C. Arguelles
  • , J. Auffenberg
  • , X. Bai
  • , M. Baker
  • , S. W. Barwick
  • , V. Baum
  • , R. Bay
  • , J. J. Beatty
  • , J. Becker Tjus
  • , K. H. Becker
  • , S. Benzvi
  • , P. Berghaus
  • , D. Berley
  • E. Bernardini, A. Bernhard, D. Z. Besson, G. Binder, D. Bindig, M. Bissok, E. Blaufuss, J. Blumenthal, D. J. Boersma, C. Bohm, D. Bose, S. Böser, O. Botner, L. Brayeur, H. P. Bretz, A. M. Brown, R. Bruijn, J. Casey, M. Casier, D. Chirkin, A. Christov, B. Christy, K. Clark, L. Classen, F. Clevermann, S. Coenders, S. Cohen, D. F. Cowen, A. H. Cruz Silva, M. Danninger, J. Daughhetee, J. C. Davis, M. Day, C. De Clercq, S. De Ridder, P. Desiati, K. D. De Vries, M. De With, T. Deyoung, J. C. Díaz-Vélez, M. Dunkman, R. Eagan, B. Eberhardt, B. Eichmann, J. Eisch, S. Euler, P. A. Evenson, O. Fadiran, A. R. Fazely, A. Fedynitch, J. Feintzeig, T. Feusels, K. Filimonov, C. Finley, T. Fischer-Wasels, S. Flis, A. Franckowiak, K. Frantzen, T. Fuchs, T. K. Gaisser, J. Gallagher, L. Gerhardt, L. Gladstone, T. Glüsenkamp, A. Goldschmidt, G. Golup, J. G. Gonzalez, J. A. Goodman, D. Góra, D. T. Grandmont, D. Grant, P. Gretskov, J. C. Groh, A. Groß, C. Ha, A. Haj Ismail, P. Hallen, A. Hallgren, F. Halzen, K. Hanson, D. Hebecker, D. Heereman, D. Heinen, K. Helbing, R. Hellauer, S. Hickford, G. C. Hill, K. D. Hoffman, R. Hoffmann, A. Homeier, K. Hoshina, F. Huang, W. Huelsnitz, P. O. Hulth, K. Hultqvist, S. Hussain, A. Ishihara, E. Jacobi, J. Jacobsen, K. Jagielski, G. S. Japaridze, K. Jero, O. Jlelati, B. Kaminsky, A. Kappes, T. Karg, A. Karle, M. Kauer, J. L. Kelley, J. Kiryluk, J. Kläs, S. R. Klein, J. H. Köhne, G. Kohnen, H. Kolanoski, L. Köpke, C. Kopper, S. Kopper, D. J. Koskinen, M. Kowalski, M. Krasberg, A. Kriesten, K. Krings, G. Kroll, J. Kunnen, N. Kurahashi, T. Kuwabara, M. Labare, H. Landsman, M. J. Larson, M. Lesiak-Bzdak, M. Leuermann, J. Leute, J. Lünemann, O. Macías, J. Madsen, G. Maggi, R. Maruyama, K. Mase, H. S. Matis, F. McNally, K. Meagher, M. Merck, G. Merino, T. Meures, S. Miarecki, E. Middell, N. Milke, J. Miller, L. Mohrmann, T. Montaruli, R. Morse, R. Nahnhauer, U. Naumann, H. Niederhausen, S. C. Nowicki, D. R. Nygren, A. Obertacke, S. Odrowski, A. Olivas, A. Omairat, A. O'Murchadha, L. Paul, J. A. Pepper, C. Pérez De Los Heros, C. Pfendner, D. Pieloth, E. Pinat, J. Posselt, P. B. Price, G. T. Przybylski, M. Quinnan, L. Rädel, I. Rae, M. Rameez, K. Rawlins, P. Redl, R. Reimann, E. Resconi, W. Rhode, M. Ribordy, M. Richman, B. Riedel, J. P. Rodrigues, C. Rott, T. Ruhe, B. Ruzybayev, D. Ryckbosch, S. M. Saba, H. G. Sander, M. Santander, S. Sarkar, K. Schatto, F. Scheriau, T. Schmidt, M. Schmitz, S. Schoenen, S. Schöneberg, A. Schönwald, A. Schukraft, L. Schulte, D. Schultz, O. Schulz, D. Seckel, Y. Sestayo, S. Seunarine, R. Shanidze, C. Sheremata, M. W.E. Smith, D. Soldin, G. M. Spiczak, C. Spiering, M. Stamatikos, T. Stanev, N. A. Stanisha, A. Stasik, T. Stezelberger, R. G. Stokstad, A. Stößl, E. A. Strahler, R. Ström, N. L. Strotjohann, G. W. Sullivan, H. Taavola, I. Taboada, A. Tamburro, A. Tepe, S. Ter-Antonyan, G. Tešić, S. Tilav, P. A. Toale, M. N. Tobin, S. Toscano, M. Tselengidou, E. Unger, M. Usner, S. Vallecorsa, N. Van Eijndhoven, A. Van Overloop, J. Van Santen, M. Vehring, M. Voge, M. Vraeghe, C. Walck, T. Waldenmaier, M. Wallraff, Ch Weaver, M. Wellons, C. Wendt, S. Westerhoff, N. Whitehorn, K. Wiebe, C. H. Wiebusch, D. R. Williams, H. Wissing, M. Wolf, T. R. Wood, K. Woschnagg, D. L. Xu, X. W. Xu, J. P. Yanez, G. Yodh, S. Yoshida, P. Zarzhitsky, J. Ziemann, S. Zierke, M. Zoll
  • University of Adelaide
  • University of Wisconsin-Madison
  • German Electron Synchrotron
  • University of Canterbury
  • University of Geneva
  • Friedrich-Alexander University Erlangen-Nürnberg
  • University of Delaware
  • South Dakota School of Mines & Technology
  • University of California at Irvine
  • Johannes Gutenberg University Mainz
  • University of California at Berkeley
  • Ohio State University
  • Ruhr University Bochum
  • University of Wuppertal
  • University of Maryland, College Park
  • Technical University of Munich
  • University of Kansas
  • Lawrence Berkeley National Laboratory
  • RWTH Aachen University
  • Uppsala University
  • Stockholm University
  • Sungkyunkwan University
  • University of Bonn
  • Vrije Universiteit Brussel
  • Swiss Federal Institute of Technology Lausanne
  • Georgia Institute of Technology
  • University of Toronto
  • TU Dortmund University
  • Pennsylvania State University
  • Ghent University
  • Humboldt University of Berlin
  • Southern University and A&M College
  • University of Alberta
  • Université libre de Bruxelles
  • Chiba University
  • Clark Atlanta University
  • Universite de Mons
  • University of Copenhagen
  • University of Alabama
  • Stony Brook University
  • University of Wisconsin-River Falls
  • National Institute for Nuclear Physics
  • University of Wisconsin
  • University of Alaska Anchorage
  • University of Oxford
  • NASA Goddard Space Flight Center

Research output: Contribution to journalArticlepeer-review

11 Scopus citations

Abstract

IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.

Original languageEnglish
Pages (from-to)198-211
Number of pages14
JournalJournal of Parallel and Distributed Computing
Volume75
DOIs
StatePublished - Jan 2015

Keywords

  • Data management
  • Distributed computing
  • Grid computing
  • Monitoring

Fingerprint

Dive into the research topics of 'The IceProd framework: Distributed data processing for the IceCube neutrino observatory'. Together they form a unique fingerprint.

Cite this