Existing Tools in HEP and Particle Astrophysics
|
|
- Jerome Patrick
- 6 years ago
- Views:
Transcription
1 Existing Tools in HEP and Particle Astrophysics Richard Dubois R.Dubois Existing Tools in HEP and Particle Astro 1/20
2 Outline Introduction: Fermi as example user Analysis Toolkits: Root and FTOOLS (et al) Shared Catalogues Data Access Code Infrastructure R.Dubois Existing Tools in HEP and Particle Astro 2/20
3 Fermi as a Case Study γ e + e HEP detector R.Dubois Existing Tools in HEP and Particle Astro 3/20
4 Fermi as a Case Study γ e + e Classic event display R.Dubois Existing Tools in HEP and Particle Astro 4/20
5 Fermi as as a a Case Study Big difference from a particle physics experiment! R.Dubois Existing Tools in HEP and Particle Astro 5/20
6 Fermi as a Case Study γ Telescope: (E, direction, time) e + R.Dubois e Existing Tools in HEP and Particle Astro 6/20
7 Reusing Software: Beg, Borrow and Steal HEP I/O format Root Event-based C++ processing framework Gaudi Many experiments make abstract layer to I/O and implement using ROOT I/O system Code management tool CMT (but moving to SCons) Detector simulation, geometry navigation GEANT4 Astro I/O format - FITS High level analysis framework & toolkit: FTOOLS Image manipulation ds9 Catalogue sharing Virtual Observatory (VO) etc High Performance Data Access xrootd R.Dubois Existing Tools in HEP and Particle Astro 7/20
8 ROOT replacement for s PAW has achieved world domination in HEP my first version was 0.96 in 1996 adopted by LHC experiments large fraction of HEP data resides in ROOT! R.Dubois Existing Tools in HEP and Particle Astro 8/20
9 ROOT feature set a floor wax and a dessert topping SNL Meant to be all inclusive if ROOT doesn t have it, you don t need it Features: I/O format and manipulation object I/O a big draw! CINT C++ interpreter C++ framework Plots, displays, graphics GUIs Geometry Significant tools Minuit (fitting) FFTs Neural nets etc etc etc R.Dubois Existing Tools in HEP and Particle Astro 9/20
10 ROOT Eye Candy R.Dubois Existing Tools in HEP and Particle Astro 10/20
11 Pros of ROOT Pro Most of the HEP world has to use it lots of experience out there A large number of dedicated users Support team is very responsive I/O format works well to describe arbitrarily complex event descriptions schema evolution and inclusion of STL objects work well Works on all platforms Many powerful 3 rd party packages included CINT mostly makes it easy to look at simple datastructures ntuples are very common format for end analysis pyroot combines power/robustness of python as scripting language with compiled ROOT classes You can do pretty much anything with it R.Dubois Existing Tools in HEP and Particle Astro 11/20
12 Cons of ROOT ROOT is now 15 years old Details of ROOT I/O format are not published and hard to reverse engineer Discourages other tools from talking to ROOT Analysis It is probably impossible to create a 100% compatible C++ interpreter, so there are bugs in CINT ROOT is for real men you can build anything you need in terms of gui s etc Essentially only very basic gui s are provided It s a lot of work to create a publication quality plot, and somewhat arcane (ie you never remember if you re not doing it all the time) No built-in interface to other data formats (except ASCII) eg FITS! Is C++ really an analysis language for mortals? Framework You must surrender to the borg R.Dubois Existing Tools in HEP and Particle Astro 12/20
13 Astro tools: FTOOLS, ds9 etc FTOOLS FITS older than ROOT early 80 s Slow to change (good and bad!) Mandated by NASA for all their x- and γ-ray missions to allow indefinite support One can fetch EGRET data with stock FTOOLS and analyse 19 yr old data! Maintained by HEASARC, with strict adherence rules Data generally is simple ntuples, tables or images Collection of utilities layered on top of FITS files and.par files Atomic operations with FITS files in/out and behaviour controlled by ASCII parameter files (or from command line or prompting) Astro folks string together lists of FTOOLS to form processing pipelines Efficient I/O functions all supplied in FTOOLS: format well documented! Write custom FTOOLS to handle the functions needed for their particular telescope Fermi wrote tools for event selection, exposure calculation, likelihood fitting all meeting the FTOOLS spec Designed for single person executing on a single box Lots of ways to shoot yourself in a batch environment R.Dubois Existing Tools in HEP and Particle Astro 13/20
14 ds9 astronomical imaging and data visualization application supports FITS images and binary tables, multiple frame buffers, region manipulation, and many scale algorithms and colormaps. provides for easy communication with external analysis tasks and is highly configurable and extensible via XPA and SAMP support for Solaris, Linux, MacOSX, and Windows supports multiple frame buffers, mosaic images, tiling, blinking, geometric markers, colormap manipulation, scaling, arbitrary zoom, rotation, pan, and a variety of coordinate systems R.Dubois Existing Tools in HEP and Particle Astro 14/20
15 Sharing Source Catalogues Astrophysics is a multi-wavelength business You can t understand a source just from one wavelength band Need access to source identification and data from other telescopes to tie up to yours Long history of catalogue search engines Allows for cross-id NED, SIMBAD are classic examples NASA is requiring its archive centers to interoperate well Now Virtual Observatories are under investigation I have no experience with them yet Markarian 421 MWL campaign R.Dubois Existing Tools in HEP and Particle Astro 15/20
16 High Performance Data Access Fermi s requirements Fermi just passed 1 PB of disk on 35 servers (mostly Sun thumpers/thors) not so much anymore clustered file system A data reprocessing is our biggest stress activity and fetches 50 TB/day from disk File system that is resistant to high loads Exploit very capable tape silo to let us tune what we keep on disk HSM with tape for quick, transparent retrieval of file Access to files outside ROOT (eg access FITS files from FTOOLS) Export as FUSE filesystem We adopted xrootd, largely due to its success with BABAR and increasing use at LHC R.Dubois Existing Tools in HEP and Particle Astro 16/20
17 xrootd in a nutshell query redirector File servers STK tape silo Distributes files across disks Maximizes throughput Minimizes manual disk management Automates archiving datasets to (and restoring from) tape Provides more reliability and scalability than NFS Remote access via proxies Supports access control based on Fermi collaborator list etc R.Dubois Existing Tools in HEP and Particle Astro 17/20
18 xrootd Scorecard Fermi usage was not like BABAR usage Many more small files; more intense usage Took a year of tuning before launch to get good performance Tools for performing some functions not so well fleshed out yet eg distributing files, file protection scheme, marking files as purgable, with a tape copy It appears we have been hit by a Solaris bug in the last year wherein heavily loaded servers drop off the network! We think it is not xrootd s fault But overall, xrootd has been a significant improvement in our intensive access to data over a collection of file servers R.Dubois Existing Tools in HEP and Particle Astro 18/20
19 Similar Data Access Beasties dcache Developed by DESY, FermiLab and Nordic Grid Castor Developed by CERN Xrootd Developed by SLAC R.Dubois Existing Tools in HEP and Particle Astro 19/20
20 Summary HEP faced the need to handle Complex detectors, particle interactions and underlying physics topologies; intensive access to data Developed tools like ROOT and xrootd Astro s needs were to handle Images and telescope data, and ensure that analysis would be possible for decades to come Developed FITS and FTOOLS ROOT has taken over the HEP world, but is getting creaky R.Dubois Existing Tools in HEP and Particle Astro 20/20
JAIDA, JAS3, WIRED4 and the AIDA tag library experience and new developments
SLAC-PUB-12950 March 2008 JAIDA, JAS3, WIRED4 and the AIDA tag library experience and new developments M Donszelmann 1, T Johnson 1, V V Serbo 1, M Turri 1 1 SLAC, 2575 Sand Hill Road, Menlo Park, CA 94025,
More informationChallenges and Evolution of the LHC Production Grid. April 13, 2011 Ian Fisk
Challenges and Evolution of the LHC Production Grid April 13, 2011 Ian Fisk 1 Evolution Uni x ALICE Remote Access PD2P/ Popularity Tier-2 Tier-2 Uni u Open Lab m Tier-2 Science Uni x Grid Uni z USA Tier-2
More informationROOT Course. Vincenzo Vitale, Dip. Fisica and INFN Roma 2
ROOT Course Vincenzo Vitale, Dip. Fisica and INFN Roma 2 Introduction This is a basic introduction to ROOT. The purpose of the course is to provide a starting knowledge and some practical experiences on
More informationHigh-Energy Physics Data-Storage Challenges
High-Energy Physics Data-Storage Challenges Richard P. Mount SLAC SC2003 Experimental HENP Understanding the quantum world requires: Repeated measurement billions of collisions Large (500 2000 physicist)
More informationA scalable storage element and its usage in HEP
AstroGrid D Meeting at MPE 14 15. November 2006 Garching dcache A scalable storage element and its usage in HEP Martin Radicke Patrick Fuhrmann Introduction to dcache 2 Project overview joint venture between
More informationPROOF-Condor integration for ATLAS
PROOF-Condor integration for ATLAS G. Ganis,, J. Iwaszkiewicz, F. Rademakers CERN / PH-SFT M. Livny, B. Mellado, Neng Xu,, Sau Lan Wu University Of Wisconsin Condor Week, Madison, 29 Apr 2 May 2008 Outline
More informationBig Computing and the Mitchell Institute for Fundamental Physics and Astronomy. David Toback
Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy Texas A&M Big Data Workshop October 2011 January 2015, Texas A&M University Research Topics Seminar 1 Outline Overview of
More informationINTRODUCTION TO THE ANAPHE/LHC++ SOFTWARE SUITE
INTRODUCTION TO THE ANAPHE/LHC++ SOFTWARE SUITE Andreas Pfeiffer CERN, Geneva, Switzerland Abstract The Anaphe/LHC++ project is an ongoing effort to provide an Object-Oriented software environment for
More informationScientific Computing at SLAC
Scientific Computing at SLAC Richard P. Mount Director: Scientific Computing and Computing Services DOE Review June 15, 2005 Scientific Computing The relationship between Science and the components of
More informationHow Can We Deliver Advanced Statistical Tools to Physicists. Ilya Narsky, Caltech
How Can We Deliver Advanced Statistical Tools to Physicists, Caltech Outline StatPatternRecognition: A C++ Package for Multivariate Classification What would be an ideal statistical framework for HEP?
More informationI Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011
I Tier-3 di CMS-Italia: stato e prospettive Claudio Grandi Workshop CCR GRID 2011 Outline INFN Perugia Tier-3 R&D Computing centre: activities, storage and batch system CMS services: bottlenecks and workarounds
More informationData Flow & Leve1 1 Pipeline
Data Flow & Leve1 1 Pipeline High level specs in L3 & L4 documents: Level 4 - LAT-SS-00505-01 Level 3 - LAT-SS-00020-01 Pipeline Server Implementation Plan - LAT-TD-00773-01 (draft in review) Database
More informationPyROOT: Seamless Melting of C++ and Python. Pere MATO, Danilo PIPARO on behalf of the ROOT Team
PyROOT: Seamless Melting of C++ and Python Pere MATO, Danilo PIPARO on behalf of the ROOT Team ROOT At the root of the experiments, project started in 1995 Open Source project (LGPL2) mainly written in
More informationSpitzer Heritage Archive
Spitzer Heritage Archive Xiuqin Wu, Trey Roby, Loi Ly IRSA/SSC, California Institute of Technology, 100-22, Pasadena, CA, USA ABSTRACT The Spitzer Heritage Archive 1 will host all the raw and final reprocessed
More informationOptimizing Parallel Access to the BaBar Database System Using CORBA Servers
SLAC-PUB-9176 September 2001 Optimizing Parallel Access to the BaBar Database System Using CORBA Servers Jacek Becla 1, Igor Gaponenko 2 1 Stanford Linear Accelerator Center Stanford University, Stanford,
More informationOperating the Distributed NDGF Tier-1
Operating the Distributed NDGF Tier-1 Michael Grønager Technical Coordinator, NDGF International Symposium on Grid Computing 08 Taipei, April 10th 2008 Talk Outline What is NDGF? Why a distributed Tier-1?
More informationSummary of Image and Plotting Software (IPS) packages collected. for possible use with GLAST Science Analysis Tools
Summary of Image and Software (IPS) packages collected for possible use with GLAST Science Analysis Tools For the Science Analysis Tools, we will need the capability to make plots, and display images on
More informationUW-ATLAS Experiences with Condor
UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS
More informationALICE ANALYSIS PRESERVATION. Mihaela Gheata DASPOS/DPHEP7 workshop
1 ALICE ANALYSIS PRESERVATION Mihaela Gheata DASPOS/DPHEP7 workshop 2 Outline ALICE data flow ALICE analysis Data & software preservation Open access and sharing analysis tools Conclusions 3 ALICE data
More informationTrivial And Non-Trivial Data Analysis for Geant4
Trivial And Non-Trivial Data Analysis for Geant4 Paul Guèye, HU Joseph Perl, SLAC 1 Simplest using text (ASCII) files Analysis Choices Geant4 does not attempt to provide its own data analysis tools, focusing
More informationirods usage at CC-IN2P3 Jean-Yves Nief
irods usage at CC-IN2P3 Jean-Yves Nief Talk overview What is CC-IN2P3? Who is using irods? irods administration: Hardware setup. irods interaction with other services: Mass Storage System, backup system,
More informationIEPSAS-Kosice: experiences in running LCG site
IEPSAS-Kosice: experiences in running LCG site Marian Babik 1, Dusan Bruncko 2, Tomas Daranyi 1, Ladislav Hluchy 1 and Pavol Strizenec 2 1 Department of Parallel and Distributed Computing, Institute of
More informationPotential use of JAS/JAIDA etc. SAS J2EE Review
Potential use of JAS/JAIDA etc. SAS J2EE Review Mark Donszelmann (standing in for Max Turri) SLAC Java Tools Group Outline Review of existing tools developed for GLAST GLAST System Tests Interface Enhanced
More informationData Analysis in ATLAS. Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser
Data Analysis in ATLAS Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser 1 ATLAS Data Flow into Analysis RAW detector data and simulated RDO data are reconstructed into our xaod
More informationGeant4 Computing Performance Benchmarking and Monitoring
Journal of Physics: Conference Series PAPER OPEN ACCESS Geant4 Computing Performance Benchmarking and Monitoring To cite this article: Andrea Dotti et al 2015 J. Phys.: Conf. Ser. 664 062021 View the article
More informationStorage Resource Sharing with CASTOR.
Storage Resource Sharing with CASTOR Olof Barring, Benjamin Couturier, Jean-Damien Durand, Emil Knezo, Sebastien Ponce (CERN) Vitali Motyakov (IHEP) ben.couturier@cern.ch 16/4/2004 Storage Resource Sharing
More informationNational Aeronautics and Space Administration Jet Propulsion Laboratory California Institute of Technology WISE Archive.
Bruce Berriman / Steve Groom Infrared Science Archive (IRSA), IPAC/Caltech GBB/SLG - 1 WSDC Functional Block Diagram White Sands JPL UCLA HRP H/K MOS Maneuvers SOC Science Team Images FTP Site Ancillary
More informationDeep Learning Photon Identification in a SuperGranular Calorimeter
Deep Learning Photon Identification in a SuperGranular Calorimeter Nikolaus Howe Maurizio Pierini Jean-Roch Vlimant @ Williams College @ CERN @ Caltech 1 Outline Introduction to the problem What is Machine
More informationSDS: A Scalable Data Services System in Data Grid
SDS: A Scalable Data s System in Data Grid Xiaoning Peng School of Information Science & Engineering, Central South University Changsha 410083, China Department of Computer Science and Technology, Huaihua
More informationScientific Computing at SLAC. Amber Boehnlein
Scientific Computing at SLAC Amber Boehnlein Amber Boehnlein Head of Scientific Computing (4/25/11) SLAC History: FNAL D0 collaboration Running experiments Department Head Simulation Department Head DOE
More informationCC-IN2P3 activity. irods in production: irods developpements in Lyon: SRB to irods migration. Hardware setup. Usage. Prospects.
Jean-Yves Nief CC-IN2P3 activity. irods in production: Hardware setup. Usage. Prospects. irods developpements in Lyon: Scripts. Micro-services. Drivers. Resource Monitoring System. icommand. SRB to irods
More informationData Centres in the Virtual Observatory Age
Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance
More informationDistributing storage of LHC data - in the nordic countries
Distributing storage of LHC data - in the nordic countries Gerd Behrmann INTEGRATE ASG Lund, May 11th, 2016 Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed
More informationFederated Data Storage System Prototype based on dcache
Federated Data Storage System Prototype based on dcache Andrey Kiryanov, Alexei Klimentov, Artem Petrosyan, Andrey Zarochentsev on behalf of BigData lab @ NRC KI and Russian Federated Data Storage Project
More informationThe Virtual Observatory and the IVOA
The Virtual Observatory and the IVOA The Virtual Observatory Emergence of the Virtual Observatory concept by 2000 Concerns about the data avalanche, with in mind in particular very large surveys such as
More informationInfluence of Distributing a Tier-2 Data Storage on Physics Analysis
ACAT Conference 2013 Influence of Distributing a Tier-2 Data Storage on Physics Analysis Jiří Horký 1,2 (horky@fzu.cz) Miloš Lokajíček 1, Jakub Peisar 2 1 Institute of Physics ASCR, 2 CESNET 17th of May,
More informationand the GridKa mass storage system Jos van Wezel / GridKa
and the GridKa mass storage system / GridKa [Tape TSM] staging server 2 Introduction Grid storage and storage middleware dcache h and TSS TSS internals Conclusion and further work 3 FZK/GridKa The GridKa
More informationLCG data management at IN2P3 CC FTS SRM dcache HPSS
jeudi 26 avril 2007 LCG data management at IN2P3 CC FTS SRM dcache HPSS Jonathan Schaeffer / Lionel Schwarz dcachemaster@cc.in2p3.fr dcache Joint development by FNAL and DESY Cache disk manager with unique
More informationWhite Paper. How the Meltdown and Spectre bugs work and what you can do to prevent a performance plummet. Contents
White Paper How the Meltdown and Spectre bugs work and what you can do to prevent a performance plummet Programs that do a lot of I/O are likely to be the worst hit by the patches designed to fix the Meltdown
More informationScientific Computing at SLAC. Chuck Boeheim Asst Director: Scientific Computing and Computing Services
Scientific Computing at SLAC Chuck Boeheim Asst Director: Scientific Computing and Computing Services 1 Drivers for SLAC Computing Computing to enable today s dataintensive science clusters, interconnects,
More informationVISPA: Visual Physics Analysis Environment
VISPA: Visual Physics Analysis Environment Tatsiana Klimkovich for the VISPA group (O.Actis, M.Erdmann, R.Fischer, A.Hinzmann, M.Kirsch, G.Müller, M.Plum, J.Steggemann) DESY Computing Seminar, 27 October
More informationData preservation for the HERA experiments at DESY using dcache technology
Journal of Physics: Conference Series PAPER OPEN ACCESS Data preservation for the HERA experiments at DESY using dcache technology To cite this article: Dirk Krücker et al 2015 J. Phys.: Conf. Ser. 66
More informationWorldwide Production Distributed Data Management at the LHC. Brian Bockelman MSST 2010, 4 May 2010
Worldwide Production Distributed Data Management at the LHC Brian Bockelman MSST 2010, 4 May 2010 At the LHC http://op-webtools.web.cern.ch/opwebtools/vistar/vistars.php?usr=lhc1 Gratuitous detector pictures:
More informationData services for LHC computing
Data services for LHC computing SLAC 1 Xavier Espinal on behalf of IT/ST DAQ to CC 8GB/s+4xReco Hot files Reliable Fast Processing DAQ Feedback loop WAN aware Tier-1/2 replica, multi-site High throughout
More informationOnline data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland
Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Abstract. The Data and Storage Services group at CERN is conducting
More informationBridging the Gap Between High Quality and High Performance for HPC Visualization
Bridging the Gap Between High Quality and High Performance for HPC Visualization Rob Sisneros National Center for Supercomputing Applications University of Illinois at Urbana Champaign Outline Why am I
More informationLong Term Data Preservation for CDF at INFN-CNAF
Long Term Data Preservation for CDF at INFN-CNAF S. Amerio 1, L. Chiarelli 2, L. dell Agnello 3, D. De Girolamo 3, D. Gregori 3, M. Pezzi 3, A. Prosperini 3, P. Ricci 3, F. Rosso 3, and S. Zani 3 1 University
More informationData Storage. Paul Millar dcache
Data Storage Paul Millar dcache Overview Introducing storage How storage is used Challenges and future directions 2 (Magnetic) Hard Disks 3 Tape systems 4 Disk enclosures 5 RAID systems 6 Types of RAID
More informationIntroduction to ROOT. M. Eads PHYS 474/790B. Friday, January 17, 14
Introduction to ROOT What is ROOT? ROOT is a software framework containing a large number of utilities useful for particle physics: More stuff than you can ever possibly need (or want)! 2 ROOT is written
More informationHEP data analysis using ROOT
HEP data analysis using ROOT week I ROOT, CLING and the command line Histograms, Graphs and Trees Mark Hodgkinson Course contents ROOT, CLING and the command line Histograms, Graphs and Trees File I/O,
More informationATLAS operations in the GridKa T1/T2 Cloud
Journal of Physics: Conference Series ATLAS operations in the GridKa T1/T2 Cloud To cite this article: G Duckeck et al 2011 J. Phys.: Conf. Ser. 331 072047 View the article online for updates and enhancements.
More informationWISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect
WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect GRITS III June 17, 2011 The Problem Heavy ops processing load R/W >3TB/day, Sustained 1Gbit/s network load 180 TB ops archive
More informationStephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S)
Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S) Overview Large Hadron Collider (LHC) Compact Muon Solenoid (CMS) experiment The Challenge Worldwide LHC
More informationData and Analysis preservation in LHCb
Data and Analysis preservation in LHCb - March 21, 2013 - S.Amerio (Padova), M.Cattaneo (CERN) Outline 2 Overview of LHCb computing model in view of long term preservation Data types and software tools
More informationIntroduction to Grid Computing
Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able
More informationTHE EUCLID ARCHIVE SYSTEM: A DATA-CENTRIC APPROACH TO BIG DATA
THE EUCLID ARCHIVE SYSTEM: A DATA-CENTRIC APPROACH TO BIG DATA Rees Williams on behalf of A.N.Belikov, D.Boxhoorn, B. Dröge, J.McFarland, A.Tsyganov, E.A. Valentijn University of Groningen, Groningen,
More informationAndrea Sciabà CERN, Switzerland
Frascati Physics Series Vol. VVVVVV (xxxx), pp. 000-000 XX Conference Location, Date-start - Date-end, Year THE LHC COMPUTING GRID Andrea Sciabà CERN, Switzerland Abstract The LHC experiments will start
More informationThe INFN Tier1. 1. INFN-CNAF, Italy
IV WORKSHOP ITALIANO SULLA FISICA DI ATLAS E CMS BOLOGNA, 23-25/11/2006 The INFN Tier1 L. dell Agnello 1), D. Bonacorsi 1), A. Chierici 1), M. Donatelli 1), A. Italiano 1), G. Lo Re 1), B. Martelli 1),
More informationLHCb Computing Resources: 2018 requests and preview of 2019 requests
LHCb Computing Resources: 2018 requests and preview of 2019 requests LHCb-PUB-2017-009 23/02/2017 LHCb Public Note Issue: 0 Revision: 0 Reference: LHCb-PUB-2017-009 Created: 23 rd February 2017 Last modified:
More informationManaging Petabytes of data with irods. Jean-Yves Nief CC-IN2P3 France
Managing Petabytes of data with irods Jean-Yves Nief CC-IN2P3 France Talk overview Data management context. Some data management goals: Storage virtualization. Virtualization of the data management policy.
More informationThe LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland
The LCG 3D Project Maria Girone, CERN The rd Open Grid Forum - OGF 4th June 2008, Barcelona Outline Introduction The Distributed Database (3D) Project Streams Replication Technology and Performance Availability
More informationRecasting with. Eric Conte, Benjamin Fuks. (Re)interpreting the results of new physics searches at the LHC June CERN
Recasting with Eric Conte, Benjamin Fuks (Re)interpreting the results of new physics searches at the LHC June 15-17 2016 @ CERN 1 Outlines 1. What is MadAnalysis 5? 2. Normal & expert mode 3. MadAnalysis
More informationPrototype DIRAC portal for EISCAT data Short instruction
Prototype DIRAC portal for EISCAT data Short instruction Carl-Fredrik Enell January 19, 2017 1 Introduction 1.1 DIRAC EGI, first European Grid Initiative, later European Grid Infrastructure, and now simply
More informationBenoit DELAUNAY Benoit DELAUNAY 1
Benoit DELAUNAY 20091023 Benoit DELAUNAY 1 CC-IN2P3 provides computing and storage for the 4 LHC experiments and many others (astro particles...) A long history of service sharing between experiments Some
More informationComputing / The DESY Grid Center
Computing / The DESY Grid Center Developing software for HEP - dcache - ILC software development The DESY Grid Center - NAF, DESY-HH and DESY-ZN Grid overview - Usage and outcome Yves Kemp for DESY IT
More informationNew data access with HTTP/WebDAV in the ATLAS experiment
New data access with HTTP/WebDAV in the ATLAS experiment Johannes Elmsheuser on behalf of the ATLAS collaboration Ludwig-Maximilians-Universität München 13 April 2015 21st International Conference on Computing
More informationArcGIS Runtime: Maximizing Performance of Your Apps. Will Jarvis and Ralf Gottschalk
ArcGIS Runtime: Maximizing Performance of Your Apps Will Jarvis and Ralf Gottschalk Agenda ArcGIS Runtime Version 100.0 Architecture How do we measure performance? We will use our internal Runtime Core
More information... Output System Layers. Application 2. Application 1. Application 3. Swing. UIKit SWT. Window System. Operating System
Output: Hardware Output System Layers Application 1 Application 2 Application 3 Swing SWT... UIKit Window System Operating System Hardware (e.g., graphics card) 2 Output Hardware 3 Start with some basics:
More informationGreece s Collaborative Ground Segment Initiatives
Greece s Collaborative Ground Segment Initiatives Dr Haris KONTOES, Prof Kanaris Tsinganos National Observatory of Athens Dr Xenofon Tsilimparis, GRNET (GEANT) The oldest Greek/SE European Research Institution:
More informationArcGIS Enterprise: Architecture & Deployment. Anthony Myers
ArcGIS Enterprise: Architecture & Deployment Anthony Myers 1 2 3 4 5 Web GIS Overview of ArcGIS Enterprise Federation & Hosted Server Deployment Patterns Implementation 1 Web GIS ArcGIS Enabling GIS for
More informationdcache: challenges and opportunities when growing into new communities Paul Millar on behalf of the dcache team
dcache: challenges and opportunities when growing into new Paul Millar communities on behalf of the dcache team EMI is partially funded by the European Commission under Grant Agreement RI-261611 Orientation:
More informationPromoting Open Standards for Digital Repository. case study examples and challenges
Promoting Open Standards for Digital Repository Infrastructures: case study examples and challenges Flavia Donno CERN P. Fuhrmann, DESY, E. Ronchieri, INFN-CNAF OGF-Europe Community Outreach Seminar Digital
More informationThe BaBar Computing Model *
SLAC PUB 9964 April 1997 The BaBar Computing Model * N. Geddes Rutherford Appleton Laboratory, Chilton, Didcot, England OX11 0QX Representing the BaBar Collaboration Abstract The BaBar experiment will
More informationTrivial Data Analysis for Geant4 Geant4 v9.3p01
Trivial Data Analysis for Geant4 Geant4 v9.3p01 Joseph Perl, SLAC 1 Simple Analysis This Week Geant4 does not attempt to provide its own data analysis tools, focusing instead on its central mission as
More informationApplication of Virtualization Technologies & CernVM. Benedikt Hegner CERN
Application of Virtualization Technologies & CernVM Benedikt Hegner CERN Virtualization Use Cases Worker Node Virtualization Software Testing Training Platform Software Deployment }Covered today Server
More informationStorage on the Lunatic Fringe. Thomas M. Ruwart University of Minnesota Digital Technology Center Intelligent Storage Consortium
Storage on the Lunatic Fringe Thomas M. Ruwart University of Minnesota Digital Technology Center Intelligent Storage Consortium tmruwart@dtc.umn.edu Orientation Who are the lunatics? What are their requirements?
More informationThe COMPASS Event Store in 2002
The COMPASS Event Store in 2002 V. Duic INFN, Trieste, Italy M. Lamanna CERN, Switzerland and INFN, Trieste, Italy COMPASS, the fixed-target experiment at CERN studying the structure of the nucleon and
More informationMulti-threaded, discrete event simulation of distributed computing systems
Multi-threaded, discrete event simulation of distributed computing systems Iosif C. Legrand California Institute of Technology, Pasadena, CA, U.S.A Abstract The LHC experiments have envisaged computing
More informationData handling and processing at the LHC experiments
1 Data handling and processing at the LHC experiments Astronomy and Bio-informatic Farida Fassi CC-IN2P3/CNRS EPAM 2011, Taza, Morocco 2 The presentation will be LHC centric, which is very relevant for
More informationScaling Without Sharding. Baron Schwartz Percona Inc Surge 2010
Scaling Without Sharding Baron Schwartz Percona Inc Surge 2010 Web Scale!!!! http://www.xtranormal.com/watch/6995033/ A Sharding Thought Experiment 64 shards per proxy [1] 1 TB of data storage per node
More informationExperiences From The Fermi Data Archive. Dr. Thomas Stephens Wyle IS/Fermi Science Support Center
Experiences From The Fermi Data Archive Dr. Thomas Stephens Wyle IS/Fermi Science Support Center A Brief Outline Fermi Mission Architecture Science Support Center Data Systems Experiences GWODWS Oct 27,
More informationEvaluation of Apache Hadoop for parallel data analysis with ROOT
Evaluation of Apache Hadoop for parallel data analysis with ROOT S Lehrack, G Duckeck, J Ebke Ludwigs-Maximilians-University Munich, Chair of elementary particle physics, Am Coulombwall 1, D-85748 Garching,
More informationExpressing Parallelism with ROOT
Expressing Parallelism with ROOT https://root.cern D. Piparo (CERN) for the ROOT team CHEP 2016 2 This Talk ROOT helps scientists to express parallelism Adopting multi-threading (MT) and multi-processing
More informationBuilding Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned. Yaroslav Tkachenko Senior Data Engineer at Activision
Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lessons Learned Yaroslav Tkachenko Senior Data Engineer at Activision 1+ PB Data lake size (AWS S3) Number of topics in the biggest
More informationFull Screen Layout. Main Menu Property-specific Options. Object Tools ( t ) Outliner. Object Properties ( n ) Properties Buttons
Object Tools ( t ) Full Screen Layout Main Menu Property-specific Options Object Properties ( n ) Properties Buttons Outliner 1 Animation Controls The Create and Add Menus 2 The Coordinate and Viewing
More informationThe Right Read Optimization is Actually Write Optimization. Leif Walsh
The Right Read Optimization is Actually Write Optimization Leif Walsh leif@tokutek.com The Right Read Optimization is Write Optimization Situation: I have some data. I want to learn things about the world,
More informationUtilizing Databases in Grid Engine 6.0
Utilizing Databases in Grid Engine 6.0 Joachim Gabler Software Engineer Sun Microsystems http://sun.com/grid Current status flat file spooling binary format for jobs ASCII format for other objects accounting
More informationTopics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
More informationGeant4 activities at DESY
Geant4 activities at DESY 3 rd Ecfa/Desy workshop Prague November 2002 Frank Gaede DESY -IT- Outline Introduction Current work Near term goals Looking ahead Conclusion ECFA/DESY workshop Prague 11/02 Frank
More informationProgramming with Python 1
Programming with Python 1 NITARP 2013: SHIPs Babar Ali Python on SHIPs 1 1 Topics Computer programs What is a program? Why program? Not useful programming Some Introductory Concepts: Components of a program
More informationEfficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library
Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library Authors Devresse Adrien (CERN) Fabrizio Furano (CERN) Typical HPC architecture Computing Cluster
More informationIcy Training - Level 1 - Introduction
Icy Training - Level 1 - Introduction Plan What is Icy? Installing Icy Graphical User Interface (GUI) Histograms & Colormap / Look up table Basic operations Overlays / Layers 3D view Icy Preferences Investigate
More informationChoosing Hardware and Operating Systems for MySQL. Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc
Choosing Hardware and Operating Systems for MySQL Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc -2- We will speak about Choosing Hardware Choosing Operating
More informationStorage and I/O requirements of the LHC experiments
Storage and I/O requirements of the LHC experiments Sverre Jarp CERN openlab, IT Dept where the Web was born 22 June 2006 OpenFabrics Workshop, Paris 1 Briefly about CERN 22 June 2006 OpenFabrics Workshop,
More informationGLAST AnaGrp 23 Jan Core Meeting Report
Core Meeting Report 17-20 Jan @ SLAC Julie, Navid, Heather, Toby, David made the trip http://wwwglast.slac.stanford.edu/software/meetingbuilder/meetingrpt.asp?mtid=3 Topics Migration to new versions of
More informationPARALLEL PROCESSING OF LARGE DATA SETS IN PARTICLE PHYSICS
PARALLEL PROCESSING OF LARGE DATA SETS IN PARTICLE PHYSICS MARINA ROTARU 1, MIHAI CIUBĂNCAN 1, GABRIEL STOICEA 1 1 Horia Hulubei National Institute for Physics and Nuclear Engineering, Reactorului 30,
More informationData Access and Data Management
Data Access and Data Management in grids Jos van Wezel Overview Background [KIT, GridKa] Practice [LHC, glite] Data storage systems [dcache a.o.] Data and meta data Intro KIT = FZK + Univ. of Karlsruhe
More informationMulti-level Translation. CS 537 Lecture 9 Paging. Example two-level page table. Multi-level Translation Analysis
Multi-level Translation CS 57 Lecture 9 Paging Michael Swift Problem: what if you have a sparse address space e.g. out of GB, you use MB spread out need one PTE per page in virtual address space bit AS
More informationRethinkDB. Niharika Vithala, Deepan Sekar, Aidan Pace, and Chang Xu
RethinkDB Niharika Vithala, Deepan Sekar, Aidan Pace, and Chang Xu Content Introduction System Features Data Model ReQL Applications Introduction Niharika Vithala What is a NoSQL Database Databases that
More informationTAP services integration at IA2 data center
TAP services integration at IA2 data center Pietro Apollo INAF - Astronomical Observatory of Trieste Outline IA2TAP: TAP implementation 2 + 1 + 1 services TapSchemaManager: Summary a supporting web application
More information