Existing Tools in HEP and Particle Astrophysics

Size: px
Start display at page:

Download "Existing Tools in HEP and Particle Astrophysics"

Transcription

1 Existing Tools in HEP and Particle Astrophysics Richard Dubois R.Dubois Existing Tools in HEP and Particle Astro 1/20

2 Outline Introduction: Fermi as example user Analysis Toolkits: Root and FTOOLS (et al) Shared Catalogues Data Access Code Infrastructure R.Dubois Existing Tools in HEP and Particle Astro 2/20

3 Fermi as a Case Study γ e + e HEP detector R.Dubois Existing Tools in HEP and Particle Astro 3/20

4 Fermi as a Case Study γ e + e Classic event display R.Dubois Existing Tools in HEP and Particle Astro 4/20

5 Fermi as as a a Case Study Big difference from a particle physics experiment! R.Dubois Existing Tools in HEP and Particle Astro 5/20

6 Fermi as a Case Study γ Telescope: (E, direction, time) e + R.Dubois e Existing Tools in HEP and Particle Astro 6/20

7 Reusing Software: Beg, Borrow and Steal HEP I/O format Root Event-based C++ processing framework Gaudi Many experiments make abstract layer to I/O and implement using ROOT I/O system Code management tool CMT (but moving to SCons) Detector simulation, geometry navigation GEANT4 Astro I/O format - FITS High level analysis framework & toolkit: FTOOLS Image manipulation ds9 Catalogue sharing Virtual Observatory (VO) etc High Performance Data Access xrootd R.Dubois Existing Tools in HEP and Particle Astro 7/20

8 ROOT replacement for s PAW has achieved world domination in HEP my first version was 0.96 in 1996 adopted by LHC experiments large fraction of HEP data resides in ROOT! R.Dubois Existing Tools in HEP and Particle Astro 8/20

9 ROOT feature set a floor wax and a dessert topping SNL Meant to be all inclusive if ROOT doesn t have it, you don t need it Features: I/O format and manipulation object I/O a big draw! CINT C++ interpreter C++ framework Plots, displays, graphics GUIs Geometry Significant tools Minuit (fitting) FFTs Neural nets etc etc etc R.Dubois Existing Tools in HEP and Particle Astro 9/20

10 ROOT Eye Candy R.Dubois Existing Tools in HEP and Particle Astro 10/20

11 Pros of ROOT Pro Most of the HEP world has to use it lots of experience out there A large number of dedicated users Support team is very responsive I/O format works well to describe arbitrarily complex event descriptions schema evolution and inclusion of STL objects work well Works on all platforms Many powerful 3 rd party packages included CINT mostly makes it easy to look at simple datastructures ntuples are very common format for end analysis pyroot combines power/robustness of python as scripting language with compiled ROOT classes You can do pretty much anything with it R.Dubois Existing Tools in HEP and Particle Astro 11/20

12 Cons of ROOT ROOT is now 15 years old Details of ROOT I/O format are not published and hard to reverse engineer Discourages other tools from talking to ROOT Analysis It is probably impossible to create a 100% compatible C++ interpreter, so there are bugs in CINT ROOT is for real men you can build anything you need in terms of gui s etc Essentially only very basic gui s are provided It s a lot of work to create a publication quality plot, and somewhat arcane (ie you never remember if you re not doing it all the time) No built-in interface to other data formats (except ASCII) eg FITS! Is C++ really an analysis language for mortals? Framework You must surrender to the borg R.Dubois Existing Tools in HEP and Particle Astro 12/20

13 Astro tools: FTOOLS, ds9 etc FTOOLS FITS older than ROOT early 80 s Slow to change (good and bad!) Mandated by NASA for all their x- and γ-ray missions to allow indefinite support One can fetch EGRET data with stock FTOOLS and analyse 19 yr old data! Maintained by HEASARC, with strict adherence rules Data generally is simple ntuples, tables or images Collection of utilities layered on top of FITS files and.par files Atomic operations with FITS files in/out and behaviour controlled by ASCII parameter files (or from command line or prompting) Astro folks string together lists of FTOOLS to form processing pipelines Efficient I/O functions all supplied in FTOOLS: format well documented! Write custom FTOOLS to handle the functions needed for their particular telescope Fermi wrote tools for event selection, exposure calculation, likelihood fitting all meeting the FTOOLS spec Designed for single person executing on a single box Lots of ways to shoot yourself in a batch environment R.Dubois Existing Tools in HEP and Particle Astro 13/20

14 ds9 astronomical imaging and data visualization application supports FITS images and binary tables, multiple frame buffers, region manipulation, and many scale algorithms and colormaps. provides for easy communication with external analysis tasks and is highly configurable and extensible via XPA and SAMP support for Solaris, Linux, MacOSX, and Windows supports multiple frame buffers, mosaic images, tiling, blinking, geometric markers, colormap manipulation, scaling, arbitrary zoom, rotation, pan, and a variety of coordinate systems R.Dubois Existing Tools in HEP and Particle Astro 14/20

15 Sharing Source Catalogues Astrophysics is a multi-wavelength business You can t understand a source just from one wavelength band Need access to source identification and data from other telescopes to tie up to yours Long history of catalogue search engines Allows for cross-id NED, SIMBAD are classic examples NASA is requiring its archive centers to interoperate well Now Virtual Observatories are under investigation I have no experience with them yet Markarian 421 MWL campaign R.Dubois Existing Tools in HEP and Particle Astro 15/20

16 High Performance Data Access Fermi s requirements Fermi just passed 1 PB of disk on 35 servers (mostly Sun thumpers/thors) not so much anymore clustered file system A data reprocessing is our biggest stress activity and fetches 50 TB/day from disk File system that is resistant to high loads Exploit very capable tape silo to let us tune what we keep on disk HSM with tape for quick, transparent retrieval of file Access to files outside ROOT (eg access FITS files from FTOOLS) Export as FUSE filesystem We adopted xrootd, largely due to its success with BABAR and increasing use at LHC R.Dubois Existing Tools in HEP and Particle Astro 16/20

17 xrootd in a nutshell query redirector File servers STK tape silo Distributes files across disks Maximizes throughput Minimizes manual disk management Automates archiving datasets to (and restoring from) tape Provides more reliability and scalability than NFS Remote access via proxies Supports access control based on Fermi collaborator list etc R.Dubois Existing Tools in HEP and Particle Astro 17/20

18 xrootd Scorecard Fermi usage was not like BABAR usage Many more small files; more intense usage Took a year of tuning before launch to get good performance Tools for performing some functions not so well fleshed out yet eg distributing files, file protection scheme, marking files as purgable, with a tape copy It appears we have been hit by a Solaris bug in the last year wherein heavily loaded servers drop off the network! We think it is not xrootd s fault But overall, xrootd has been a significant improvement in our intensive access to data over a collection of file servers R.Dubois Existing Tools in HEP and Particle Astro 18/20

19 Similar Data Access Beasties dcache Developed by DESY, FermiLab and Nordic Grid Castor Developed by CERN Xrootd Developed by SLAC R.Dubois Existing Tools in HEP and Particle Astro 19/20

20 Summary HEP faced the need to handle Complex detectors, particle interactions and underlying physics topologies; intensive access to data Developed tools like ROOT and xrootd Astro s needs were to handle Images and telescope data, and ensure that analysis would be possible for decades to come Developed FITS and FTOOLS ROOT has taken over the HEP world, but is getting creaky R.Dubois Existing Tools in HEP and Particle Astro 20/20

JAIDA, JAS3, WIRED4 and the AIDA tag library experience and new developments

JAIDA, JAS3, WIRED4 and the AIDA tag library experience and new developments SLAC-PUB-12950 March 2008 JAIDA, JAS3, WIRED4 and the AIDA tag library experience and new developments M Donszelmann 1, T Johnson 1, V V Serbo 1, M Turri 1 1 SLAC, 2575 Sand Hill Road, Menlo Park, CA 94025,

More information

Challenges and Evolution of the LHC Production Grid. April 13, 2011 Ian Fisk

Challenges and Evolution of the LHC Production Grid. April 13, 2011 Ian Fisk Challenges and Evolution of the LHC Production Grid April 13, 2011 Ian Fisk 1 Evolution Uni x ALICE Remote Access PD2P/ Popularity Tier-2 Tier-2 Uni u Open Lab m Tier-2 Science Uni x Grid Uni z USA Tier-2

More information

ROOT Course. Vincenzo Vitale, Dip. Fisica and INFN Roma 2

ROOT Course. Vincenzo Vitale, Dip. Fisica and INFN Roma 2 ROOT Course Vincenzo Vitale, Dip. Fisica and INFN Roma 2 Introduction This is a basic introduction to ROOT. The purpose of the course is to provide a starting knowledge and some practical experiences on

More information

High-Energy Physics Data-Storage Challenges

High-Energy Physics Data-Storage Challenges High-Energy Physics Data-Storage Challenges Richard P. Mount SLAC SC2003 Experimental HENP Understanding the quantum world requires: Repeated measurement billions of collisions Large (500 2000 physicist)

More information

A scalable storage element and its usage in HEP

A scalable storage element and its usage in HEP AstroGrid D Meeting at MPE 14 15. November 2006 Garching dcache A scalable storage element and its usage in HEP Martin Radicke Patrick Fuhrmann Introduction to dcache 2 Project overview joint venture between

More information

PROOF-Condor integration for ATLAS

PROOF-Condor integration for ATLAS PROOF-Condor integration for ATLAS G. Ganis,, J. Iwaszkiewicz, F. Rademakers CERN / PH-SFT M. Livny, B. Mellado, Neng Xu,, Sau Lan Wu University Of Wisconsin Condor Week, Madison, 29 Apr 2 May 2008 Outline

More information

Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy. David Toback

Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy. David Toback Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy Texas A&M Big Data Workshop October 2011 January 2015, Texas A&M University Research Topics Seminar 1 Outline Overview of

More information

INTRODUCTION TO THE ANAPHE/LHC++ SOFTWARE SUITE

INTRODUCTION TO THE ANAPHE/LHC++ SOFTWARE SUITE INTRODUCTION TO THE ANAPHE/LHC++ SOFTWARE SUITE Andreas Pfeiffer CERN, Geneva, Switzerland Abstract The Anaphe/LHC++ project is an ongoing effort to provide an Object-Oriented software environment for

More information

Scientific Computing at SLAC

Scientific Computing at SLAC Scientific Computing at SLAC Richard P. Mount Director: Scientific Computing and Computing Services DOE Review June 15, 2005 Scientific Computing The relationship between Science and the components of

More information

How Can We Deliver Advanced Statistical Tools to Physicists. Ilya Narsky, Caltech

How Can We Deliver Advanced Statistical Tools to Physicists. Ilya Narsky, Caltech How Can We Deliver Advanced Statistical Tools to Physicists, Caltech Outline StatPatternRecognition: A C++ Package for Multivariate Classification What would be an ideal statistical framework for HEP?

More information

I Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011

I Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011 I Tier-3 di CMS-Italia: stato e prospettive Claudio Grandi Workshop CCR GRID 2011 Outline INFN Perugia Tier-3 R&D Computing centre: activities, storage and batch system CMS services: bottlenecks and workarounds

More information

Data Flow & Leve1 1 Pipeline

Data Flow & Leve1 1 Pipeline Data Flow & Leve1 1 Pipeline High level specs in L3 & L4 documents: Level 4 - LAT-SS-00505-01 Level 3 - LAT-SS-00020-01 Pipeline Server Implementation Plan - LAT-TD-00773-01 (draft in review) Database

More information

PyROOT: Seamless Melting of C++ and Python. Pere MATO, Danilo PIPARO on behalf of the ROOT Team

PyROOT: Seamless Melting of C++ and Python. Pere MATO, Danilo PIPARO on behalf of the ROOT Team PyROOT: Seamless Melting of C++ and Python Pere MATO, Danilo PIPARO on behalf of the ROOT Team ROOT At the root of the experiments, project started in 1995 Open Source project (LGPL2) mainly written in

More information

Spitzer Heritage Archive

Spitzer Heritage Archive Spitzer Heritage Archive Xiuqin Wu, Trey Roby, Loi Ly IRSA/SSC, California Institute of Technology, 100-22, Pasadena, CA, USA ABSTRACT The Spitzer Heritage Archive 1 will host all the raw and final reprocessed

More information

Optimizing Parallel Access to the BaBar Database System Using CORBA Servers

Optimizing Parallel Access to the BaBar Database System Using CORBA Servers SLAC-PUB-9176 September 2001 Optimizing Parallel Access to the BaBar Database System Using CORBA Servers Jacek Becla 1, Igor Gaponenko 2 1 Stanford Linear Accelerator Center Stanford University, Stanford,

More information

Operating the Distributed NDGF Tier-1

Operating the Distributed NDGF Tier-1 Operating the Distributed NDGF Tier-1 Michael Grønager Technical Coordinator, NDGF International Symposium on Grid Computing 08 Taipei, April 10th 2008 Talk Outline What is NDGF? Why a distributed Tier-1?

More information

Summary of Image and Plotting Software (IPS) packages collected. for possible use with GLAST Science Analysis Tools

Summary of Image and Plotting Software (IPS) packages collected. for possible use with GLAST Science Analysis Tools Summary of Image and Software (IPS) packages collected for possible use with GLAST Science Analysis Tools For the Science Analysis Tools, we will need the capability to make plots, and display images on

More information

UW-ATLAS Experiences with Condor

UW-ATLAS Experiences with Condor UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS

More information

ALICE ANALYSIS PRESERVATION. Mihaela Gheata DASPOS/DPHEP7 workshop

ALICE ANALYSIS PRESERVATION. Mihaela Gheata DASPOS/DPHEP7 workshop 1 ALICE ANALYSIS PRESERVATION Mihaela Gheata DASPOS/DPHEP7 workshop 2 Outline ALICE data flow ALICE analysis Data & software preservation Open access and sharing analysis tools Conclusions 3 ALICE data

More information

Trivial And Non-Trivial Data Analysis for Geant4

Trivial And Non-Trivial Data Analysis for Geant4 Trivial And Non-Trivial Data Analysis for Geant4 Paul Guèye, HU Joseph Perl, SLAC 1 Simplest using text (ASCII) files Analysis Choices Geant4 does not attempt to provide its own data analysis tools, focusing

More information

irods usage at CC-IN2P3 Jean-Yves Nief

irods usage at CC-IN2P3 Jean-Yves Nief irods usage at CC-IN2P3 Jean-Yves Nief Talk overview What is CC-IN2P3? Who is using irods? irods administration: Hardware setup. irods interaction with other services: Mass Storage System, backup system,

More information

IEPSAS-Kosice: experiences in running LCG site

IEPSAS-Kosice: experiences in running LCG site IEPSAS-Kosice: experiences in running LCG site Marian Babik 1, Dusan Bruncko 2, Tomas Daranyi 1, Ladislav Hluchy 1 and Pavol Strizenec 2 1 Department of Parallel and Distributed Computing, Institute of

More information

Potential use of JAS/JAIDA etc. SAS J2EE Review

Potential use of JAS/JAIDA etc. SAS J2EE Review Potential use of JAS/JAIDA etc. SAS J2EE Review Mark Donszelmann (standing in for Max Turri) SLAC Java Tools Group Outline Review of existing tools developed for GLAST GLAST System Tests Interface Enhanced

More information

Data Analysis in ATLAS. Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser

Data Analysis in ATLAS. Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser Data Analysis in ATLAS Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser 1 ATLAS Data Flow into Analysis RAW detector data and simulated RDO data are reconstructed into our xaod

More information

Geant4 Computing Performance Benchmarking and Monitoring

Geant4 Computing Performance Benchmarking and Monitoring Journal of Physics: Conference Series PAPER OPEN ACCESS Geant4 Computing Performance Benchmarking and Monitoring To cite this article: Andrea Dotti et al 2015 J. Phys.: Conf. Ser. 664 062021 View the article

More information

Storage Resource Sharing with CASTOR.

Storage Resource Sharing with CASTOR. Storage Resource Sharing with CASTOR Olof Barring, Benjamin Couturier, Jean-Damien Durand, Emil Knezo, Sebastien Ponce (CERN) Vitali Motyakov (IHEP) ben.couturier@cern.ch 16/4/2004 Storage Resource Sharing

More information

National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive.

National Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive. Bruce Berriman / Steve Groom Infrared Science Archive (IRSA), IPAC/Caltech GBB/SLG - 1 WSDC Functional Block Diagram White Sands JPL UCLA HRP H/K MOS Maneuvers SOC Science Team Images FTP Site Ancillary

More information

Deep Learning Photon Identification in a SuperGranular Calorimeter

Deep Learning Photon Identification in a SuperGranular Calorimeter Deep Learning Photon Identification in a SuperGranular Calorimeter Nikolaus Howe Maurizio Pierini Jean-Roch Vlimant @ Williams College @ CERN @ Caltech 1 Outline Introduction to the problem What is Machine

More information

SDS: A Scalable Data Services System in Data Grid

SDS: A Scalable Data Services System in Data Grid SDS: A Scalable Data s System in Data Grid Xiaoning Peng School of Information Science & Engineering, Central South University Changsha 410083, China Department of Computer Science and Technology, Huaihua

More information

Scientific Computing at SLAC. Amber Boehnlein

Scientific Computing at SLAC. Amber Boehnlein Scientific Computing at SLAC Amber Boehnlein Amber Boehnlein Head of Scientific Computing (4/25/11) SLAC History: FNAL D0 collaboration Running experiments Department Head Simulation Department Head DOE

More information

CC-IN2P3 activity. irods in production: irods developpements in Lyon: SRB to irods migration. Hardware setup. Usage. Prospects.

CC-IN2P3 activity. irods in production: irods developpements in Lyon: SRB to irods migration. Hardware setup. Usage. Prospects. Jean-Yves Nief CC-IN2P3 activity. irods in production: Hardware setup. Usage. Prospects. irods developpements in Lyon: Scripts. Micro-services. Drivers. Resource Monitoring System. icommand. SRB to irods

More information

Data Centres in the Virtual Observatory Age

Data Centres in the Virtual Observatory Age Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance

More information

Distributing storage of LHC data - in the nordic countries

Distributing storage of LHC data - in the nordic countries Distributing storage of LHC data - in the nordic countries Gerd Behrmann INTEGRATE ASG Lund, May 11th, 2016 Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed

More information

Federated Data Storage System Prototype based on dcache

Federated Data Storage System Prototype based on dcache Federated Data Storage System Prototype based on dcache Andrey Kiryanov, Alexei Klimentov, Artem Petrosyan, Andrey Zarochentsev on behalf of BigData lab @ NRC KI and Russian Federated Data Storage Project

More information

The Virtual Observatory and the IVOA

The Virtual Observatory and the IVOA The Virtual Observatory and the IVOA The Virtual Observatory Emergence of the Virtual Observatory concept by 2000 Concerns about the data avalanche, with in mind in particular very large surveys such as

More information

Influence of Distributing a Tier-2 Data Storage on Physics Analysis

Influence of Distributing a Tier-2 Data Storage on Physics Analysis ACAT Conference 2013 Influence of Distributing a Tier-2 Data Storage on Physics Analysis Jiří Horký 1,2 (horky@fzu.cz) Miloš Lokajíček 1, Jakub Peisar 2 1 Institute of Physics ASCR, 2 CESNET 17th of May,

More information

and the GridKa mass storage system Jos van Wezel / GridKa

and the GridKa mass storage system Jos van Wezel / GridKa and the GridKa mass storage system / GridKa [Tape TSM] staging server 2 Introduction Grid storage and storage middleware dcache h and TSS TSS internals Conclusion and further work 3 FZK/GridKa The GridKa

More information

LCG data management at IN2P3 CC FTS SRM dcache HPSS

LCG data management at IN2P3 CC FTS SRM dcache HPSS jeudi 26 avril 2007 LCG data management at IN2P3 CC FTS SRM dcache HPSS Jonathan Schaeffer / Lionel Schwarz dcachemaster@cc.in2p3.fr dcache Joint development by FNAL and DESY Cache disk manager with unique

More information

White Paper. How the Meltdown and Spectre bugs work and what you can do to prevent a performance plummet. Contents

White Paper. How the Meltdown and Spectre bugs work and what you can do to prevent a performance plummet. Contents White Paper How the Meltdown and Spectre bugs work and what you can do to prevent a performance plummet Programs that do a lot of I/O are likely to be the worst hit by the patches designed to fix the Meltdown

More information

Scientific Computing at SLAC. Chuck Boeheim Asst Director: Scientific Computing and Computing Services

Scientific Computing at SLAC. Chuck Boeheim Asst Director: Scientific Computing and Computing Services Scientific Computing at SLAC Chuck Boeheim Asst Director: Scientific Computing and Computing Services 1 Drivers for SLAC Computing Computing to enable today s dataintensive science clusters, interconnects,

More information

VISPA: Visual Physics Analysis Environment

VISPA: Visual Physics Analysis Environment VISPA: Visual Physics Analysis Environment Tatsiana Klimkovich for the VISPA group (O.Actis, M.Erdmann, R.Fischer, A.Hinzmann, M.Kirsch, G.Müller, M.Plum, J.Steggemann) DESY Computing Seminar, 27 October

More information

Data preservation for the HERA experiments at DESY using dcache technology

Data preservation for the HERA experiments at DESY using dcache technology Journal of Physics: Conference Series PAPER OPEN ACCESS Data preservation for the HERA experiments at DESY using dcache technology To cite this article: Dirk Krücker et al 2015 J. Phys.: Conf. Ser. 66

More information

Worldwide Production Distributed Data Management at the LHC. Brian Bockelman MSST 2010, 4 May 2010

Worldwide Production Distributed Data Management at the LHC. Brian Bockelman MSST 2010, 4 May 2010 Worldwide Production Distributed Data Management at the LHC Brian Bockelman MSST 2010, 4 May 2010 At the LHC http://op-webtools.web.cern.ch/opwebtools/vistar/vistars.php?usr=lhc1 Gratuitous detector pictures:

More information

Data services for LHC computing

Data services for LHC computing Data services for LHC computing SLAC 1 Xavier Espinal on behalf of IT/ST DAQ to CC 8GB/s+4xReco Hot files Reliable Fast Processing DAQ Feedback loop WAN aware Tier-1/2 replica, multi-site High throughout

More information

Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland

Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Abstract. The Data and Storage Services group at CERN is conducting

More information

Bridging the Gap Between High Quality and High Performance for HPC Visualization

Bridging the Gap Between High Quality and High Performance for HPC Visualization Bridging the Gap Between High Quality and High Performance for HPC Visualization Rob Sisneros National Center for Supercomputing Applications University of Illinois at Urbana Champaign Outline Why am I

More information

Long Term Data Preservation for CDF at INFN-CNAF

Long Term Data Preservation for CDF at INFN-CNAF Long Term Data Preservation for CDF at INFN-CNAF S. Amerio 1, L. Chiarelli 2, L. dell Agnello 3, D. De Girolamo 3, D. Gregori 3, M. Pezzi 3, A. Prosperini 3, P. Ricci 3, F. Rosso 3, and S. Zani 3 1 University

More information

Data Storage. Paul Millar dcache

Data Storage. Paul Millar dcache Data Storage Paul Millar dcache Overview Introducing storage How storage is used Challenges and future directions 2 (Magnetic) Hard Disks 3 Tape systems 4 Disk enclosures 5 RAID systems 6 Types of RAID

More information

Introduction to ROOT. M. Eads PHYS 474/790B. Friday, January 17, 14

Introduction to ROOT. M. Eads PHYS 474/790B. Friday, January 17, 14 Introduction to ROOT What is ROOT? ROOT is a software framework containing a large number of utilities useful for particle physics: More stuff than you can ever possibly need (or want)! 2 ROOT is written

More information

HEP data analysis using ROOT

HEP data analysis using ROOT HEP data analysis using ROOT week I ROOT, CLING and the command line Histograms, Graphs and Trees Mark Hodgkinson Course contents ROOT, CLING and the command line Histograms, Graphs and Trees File I/O,

More information

ATLAS operations in the GridKa T1/T2 Cloud

ATLAS operations in the GridKa T1/T2 Cloud Journal of Physics: Conference Series ATLAS operations in the GridKa T1/T2 Cloud To cite this article: G Duckeck et al 2011 J. Phys.: Conf. Ser. 331 072047 View the article online for updates and enhancements.

More information

WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect

WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect GRITS III June 17, 2011 The Problem Heavy ops processing load R/W >3TB/day, Sustained 1Gbit/s network load 180 TB ops archive

More information

Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S)

Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S) Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S) Overview Large Hadron Collider (LHC) Compact Muon Solenoid (CMS) experiment The Challenge Worldwide LHC

More information

Data and Analysis preservation in LHCb

Data and Analysis preservation in LHCb Data and Analysis preservation in LHCb - March 21, 2013 - S.Amerio (Padova), M.Cattaneo (CERN) Outline 2 Overview of LHCb computing model in view of long term preservation Data types and software tools

More information

Introduction to Grid Computing

Introduction to Grid Computing Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able

More information

THE EUCLID ARCHIVE SYSTEM: A DATA-CENTRIC APPROACH TO BIG DATA

THE EUCLID ARCHIVE SYSTEM: A DATA-CENTRIC APPROACH TO BIG DATA THE EUCLID ARCHIVE SYSTEM: A DATA-CENTRIC APPROACH TO BIG DATA Rees Williams on behalf of A.N.Belikov, D.Boxhoorn, B. Dröge, J.McFarland, A.Tsyganov, E.A. Valentijn University of Groningen, Groningen,

More information

Andrea Sciabà CERN, Switzerland

Andrea Sciabà CERN, Switzerland Frascati Physics Series Vol. VVVVVV (xxxx), pp. 000-000 XX Conference Location, Date-start - Date-end, Year THE LHC COMPUTING GRID Andrea Sciabà CERN, Switzerland Abstract The LHC experiments will start

More information

The INFN Tier1. 1. INFN-CNAF, Italy

The INFN Tier1. 1. INFN-CNAF, Italy IV WORKSHOP ITALIANO SULLA FISICA DI ATLAS E CMS BOLOGNA, 23-25/11/2006 The INFN Tier1 L. dell Agnello 1), D. Bonacorsi 1), A. Chierici 1), M. Donatelli 1), A. Italiano 1), G. Lo Re 1), B. Martelli 1),

More information

LHCb Computing Resources: 2018 requests and preview of 2019 requests

LHCb Computing Resources: 2018 requests and preview of 2019 requests LHCb Computing Resources: 2018 requests and preview of 2019 requests LHCb-PUB-2017-009 23/02/2017 LHCb Public Note Issue: 0 Revision: 0 Reference: LHCb-PUB-2017-009 Created: 23 rd February 2017 Last modified:

More information

Managing Petabytes of data with irods. Jean-Yves Nief CC-IN2P3 France

Managing Petabytes of data with irods. Jean-Yves Nief CC-IN2P3 France Managing Petabytes of data with irods Jean-Yves Nief CC-IN2P3 France Talk overview Data management context. Some data management goals: Storage virtualization. Virtualization of the data management policy.

More information

The LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland

The LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland The LCG 3D Project Maria Girone, CERN The rd Open Grid Forum - OGF 4th June 2008, Barcelona Outline Introduction The Distributed Database (3D) Project Streams Replication Technology and Performance Availability

More information

Recasting with. Eric Conte, Benjamin Fuks. (Re)interpreting the results of new physics searches at the LHC June CERN

Recasting with. Eric Conte, Benjamin Fuks. (Re)interpreting the results of new physics searches at the LHC June CERN Recasting with Eric Conte, Benjamin Fuks (Re)interpreting the results of new physics searches at the LHC June 15-17 2016 @ CERN 1 Outlines 1. What is MadAnalysis 5? 2. Normal & expert mode 3. MadAnalysis

More information

Prototype DIRAC portal for EISCAT data Short instruction

Prototype DIRAC portal for EISCAT data Short instruction Prototype DIRAC portal for EISCAT data Short instruction Carl-Fredrik Enell January 19, 2017 1 Introduction 1.1 DIRAC EGI, first European Grid Initiative, later European Grid Infrastructure, and now simply

More information

Benoit DELAUNAY Benoit DELAUNAY 1

Benoit DELAUNAY Benoit DELAUNAY 1 Benoit DELAUNAY 20091023 Benoit DELAUNAY 1 CC-IN2P3 provides computing and storage for the 4 LHC experiments and many others (astro particles...) A long history of service sharing between experiments Some

More information

Computing / The DESY Grid Center

Computing / The DESY Grid Center Computing / The DESY Grid Center Developing software for HEP - dcache - ILC software development The DESY Grid Center - NAF, DESY-HH and DESY-ZN Grid overview - Usage and outcome Yves Kemp for DESY IT

More information

New data access with HTTP/WebDAV in the ATLAS experiment

New data access with HTTP/WebDAV in the ATLAS experiment New data access with HTTP/WebDAV in the ATLAS experiment Johannes Elmsheuser on behalf of the ATLAS collaboration Ludwig-Maximilians-Universität München 13 April 2015 21st International Conference on Computing

More information

ArcGIS Runtime: Maximizing Performance of Your Apps. Will Jarvis and Ralf Gottschalk

ArcGIS Runtime: Maximizing Performance of Your Apps. Will Jarvis and Ralf Gottschalk ArcGIS Runtime: Maximizing Performance of Your Apps Will Jarvis and Ralf Gottschalk Agenda ArcGIS Runtime Version 100.0 Architecture How do we measure performance? We will use our internal Runtime Core

More information

... Output System Layers. Application 2. Application 1. Application 3. Swing. UIKit SWT. Window System. Operating System

... Output System Layers. Application 2. Application 1. Application 3. Swing. UIKit SWT. Window System. Operating System Output: Hardware Output System Layers Application 1 Application 2 Application 3 Swing SWT... UIKit Window System Operating System Hardware (e.g., graphics card) 2 Output Hardware 3 Start with some basics:

More information

Greece s Collaborative Ground Segment Initiatives

Greece s Collaborative Ground Segment Initiatives Greece s Collaborative Ground Segment Initiatives Dr Haris KONTOES, Prof Kanaris Tsinganos National Observatory of Athens Dr Xenofon Tsilimparis, GRNET (GEANT) The oldest Greek/SE European Research Institution:

More information

ArcGIS Enterprise: Architecture & Deployment. Anthony Myers

ArcGIS Enterprise: Architecture & Deployment. Anthony Myers ArcGIS Enterprise: Architecture & Deployment Anthony Myers 1 2 3 4 5 Web GIS Overview of ArcGIS Enterprise Federation & Hosted Server Deployment Patterns Implementation 1 Web GIS ArcGIS Enabling GIS for

More information

dcache: challenges and opportunities when growing into new communities Paul Millar on behalf of the dcache team

dcache: challenges and opportunities when growing into new communities Paul Millar on behalf of the dcache team dcache: challenges and opportunities when growing into new Paul Millar communities on behalf of the dcache team EMI is partially funded by the European Commission under Grant Agreement RI-261611 Orientation:

More information

Promoting Open Standards for Digital Repository. case study examples and challenges

Promoting Open Standards for Digital Repository. case study examples and challenges Promoting Open Standards for Digital Repository Infrastructures: case study examples and challenges Flavia Donno CERN P. Fuhrmann, DESY, E. Ronchieri, INFN-CNAF OGF-Europe Community Outreach Seminar Digital

More information

The BaBar Computing Model *

The BaBar Computing Model * SLAC PUB 9964 April 1997 The BaBar Computing Model * N. Geddes Rutherford Appleton Laboratory, Chilton, Didcot, England OX11 0QX Representing the BaBar Collaboration Abstract The BaBar experiment will

More information

Trivial Data Analysis for Geant4 Geant4 v9.3p01

Trivial Data Analysis for Geant4 Geant4 v9.3p01 Trivial Data Analysis for Geant4 Geant4 v9.3p01 Joseph Perl, SLAC 1 Simple Analysis This Week Geant4 does not attempt to provide its own data analysis tools, focusing instead on its central mission as

More information

Application of Virtualization Technologies & CernVM. Benedikt Hegner CERN

Application of Virtualization Technologies & CernVM. Benedikt Hegner CERN Application of Virtualization Technologies & CernVM Benedikt Hegner CERN Virtualization Use Cases Worker Node Virtualization Software Testing Training Platform Software Deployment }Covered today Server

More information

Storage on the Lunatic Fringe. Thomas M. Ruwart University of Minnesota Digital Technology Center Intelligent Storage Consortium

Storage on the Lunatic Fringe. Thomas M. Ruwart University of Minnesota Digital Technology Center Intelligent Storage Consortium Storage on the Lunatic Fringe Thomas M. Ruwart University of Minnesota Digital Technology Center Intelligent Storage Consortium tmruwart@dtc.umn.edu Orientation Who are the lunatics? What are their requirements?

More information

The COMPASS Event Store in 2002

The COMPASS Event Store in 2002 The COMPASS Event Store in 2002 V. Duic INFN, Trieste, Italy M. Lamanna CERN, Switzerland and INFN, Trieste, Italy COMPASS, the fixed-target experiment at CERN studying the structure of the nucleon and

More information

Multi-threaded, discrete event simulation of distributed computing systems

Multi-threaded, discrete event simulation of distributed computing systems Multi-threaded, discrete event simulation of distributed computing systems Iosif C. Legrand California Institute of Technology, Pasadena, CA, U.S.A Abstract The LHC experiments have envisaged computing

More information

Data handling and processing at the LHC experiments

Data handling and processing at the LHC experiments 1 Data handling and processing at the LHC experiments Astronomy and Bio-informatic Farida Fassi CC-IN2P3/CNRS EPAM 2011, Taza, Morocco 2 The presentation will be LHC centric, which is very relevant for

More information

Scaling Without Sharding. Baron Schwartz Percona Inc Surge 2010

Scaling Without Sharding. Baron Schwartz Percona Inc Surge 2010 Scaling Without Sharding Baron Schwartz Percona Inc Surge 2010 Web Scale!!!! http://www.xtranormal.com/watch/6995033/ A Sharding Thought Experiment 64 shards per proxy [1] 1 TB of data storage per node

More information

Experiences From The Fermi Data Archive. Dr. Thomas Stephens Wyle IS/Fermi Science Support Center

Experiences From The Fermi Data Archive. Dr. Thomas Stephens Wyle IS/Fermi Science Support Center Experiences From The Fermi Data Archive Dr. Thomas Stephens Wyle IS/Fermi Science Support Center A Brief Outline Fermi Mission Architecture Science Support Center Data Systems Experiences GWODWS Oct 27,

More information

Evaluation of Apache Hadoop for parallel data analysis with ROOT

Evaluation of Apache Hadoop for parallel data analysis with ROOT Evaluation of Apache Hadoop for parallel data analysis with ROOT S Lehrack, G Duckeck, J Ebke Ludwigs-Maximilians-University Munich, Chair of elementary particle physics, Am Coulombwall 1, D-85748 Garching,

More information

Expressing Parallelism with ROOT

Expressing Parallelism with ROOT Expressing Parallelism with ROOT https://root.cern D. Piparo (CERN) for the ROOT team CHEP 2016 2 This Talk ROOT helps scientists to express parallelism Adopting multi-threading (MT) and multi-processing

More information

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned. Yaroslav Tkachenko Senior Data Engineer at Activision

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned. Yaroslav Tkachenko Senior Data Engineer at Activision Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav Tkachenko Senior Data Engineer at Activision 1+ PB Data lake size (AWS S3) Number of topics in the biggest

More information

Full Screen Layout. Main Menu Property-specific Options. Object Tools ( t ) Outliner. Object Properties ( n ) Properties Buttons

Full Screen Layout. Main Menu Property-specific Options. Object Tools ( t ) Outliner. Object Properties ( n ) Properties Buttons Object Tools ( t ) Full Screen Layout Main Menu Property-specific Options Object Properties ( n ) Properties Buttons Outliner 1 Animation Controls The Create and Add Menus 2 The Coordinate and Viewing

More information

The Right Read Optimization is Actually Write Optimization. Leif Walsh

The Right Read Optimization is Actually Write Optimization. Leif Walsh The Right Read Optimization is Actually Write Optimization Leif Walsh leif@tokutek.com The Right Read Optimization is Write Optimization Situation: I have some data. I want to learn things about the world,

More information

Utilizing Databases in Grid Engine 6.0

Utilizing Databases in Grid Engine 6.0 Utilizing Databases in Grid Engine 6.0 Joachim Gabler Software Engineer Sun Microsystems http://sun.com/grid Current status flat file spooling binary format for jobs ASCII format for other objects accounting

More information

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples

Topics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?

More information

Geant4 activities at DESY

Geant4 activities at DESY Geant4 activities at DESY 3 rd Ecfa/Desy workshop Prague November 2002 Frank Gaede DESY -IT- Outline Introduction Current work Near term goals Looking ahead Conclusion ECFA/DESY workshop Prague 11/02 Frank

More information

Programming with Python 1

Programming with Python 1 Programming with Python 1 NITARP 2013: SHIPs Babar Ali Python on SHIPs 1 1 Topics Computer programs What is a program? Why program? Not useful programming Some Introductory Concepts: Components of a program

More information

Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library

Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library Authors Devresse Adrien (CERN) Fabrizio Furano (CERN) Typical HPC architecture Computing Cluster

More information

Icy Training - Level 1 - Introduction

Icy Training - Level 1 - Introduction Icy Training - Level 1 - Introduction Plan What is Icy? Installing Icy Graphical User Interface (GUI) Histograms & Colormap / Look up table Basic operations Overlays / Layers 3D view Icy Preferences Investigate

More information

Choosing Hardware and Operating Systems for MySQL. Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc

Choosing Hardware and Operating Systems for MySQL. Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc Choosing Hardware and Operating Systems for MySQL Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc -2- We will speak about Choosing Hardware Choosing Operating

More information

Storage and I/O requirements of the LHC experiments

Storage and I/O requirements of the LHC experiments Storage and I/O requirements of the LHC experiments Sverre Jarp CERN openlab, IT Dept where the Web was born 22 June 2006 OpenFabrics Workshop, Paris 1 Briefly about CERN 22 June 2006 OpenFabrics Workshop,

More information

GLAST AnaGrp 23 Jan Core Meeting Report

GLAST AnaGrp 23 Jan Core Meeting Report Core Meeting Report 17-20 Jan @ SLAC Julie, Navid, Heather, Toby, David made the trip http://wwwglast.slac.stanford.edu/software/meetingbuilder/meetingrpt.asp?mtid=3 Topics Migration to new versions of

More information

PARALLEL PROCESSING OF LARGE DATA SETS IN PARTICLE PHYSICS

PARALLEL PROCESSING OF LARGE DATA SETS IN PARTICLE PHYSICS PARALLEL PROCESSING OF LARGE DATA SETS IN PARTICLE PHYSICS MARINA ROTARU 1, MIHAI CIUBĂNCAN 1, GABRIEL STOICEA 1 1 Horia Hulubei National Institute for Physics and Nuclear Engineering, Reactorului 30,

More information

Data Access and Data Management

Data Access and Data Management Data Access and Data Management in grids Jos van Wezel Overview Background [KIT, GridKa] Practice [LHC, glite] Data storage systems [dcache a.o.] Data and meta data Intro KIT = FZK + Univ. of Karlsruhe

More information

Multi-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis

Multi-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis Multi-level Translation CS 57 Lecture 9 Paging Michael Swift Problem: what if you have a sparse address space e.g. out of GB, you use MB spread out need one PTE per page in virtual address space bit AS

More information

RethinkDB. Niharika Vithala, Deepan Sekar, Aidan Pace, and Chang Xu

RethinkDB. Niharika Vithala, Deepan Sekar, Aidan Pace, and Chang Xu RethinkDB Niharika Vithala, Deepan Sekar, Aidan Pace, and Chang Xu Content Introduction System Features Data Model ReQL Applications Introduction Niharika Vithala What is a NoSQL Database Databases that

More information

TAP services integration at IA2 data center

TAP services integration at IA2 data center TAP services integration at IA2 data center Pietro Apollo INAF - Astronomical Observatory of Trieste Outline IA2TAP: TAP implementation 2 + 1 + 1 services TapSchemaManager: Summary a supporting web application

More information