The ARDA project: Grid analysis prototypes of the LHC experiments

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "The ARDA project: Grid analysis prototypes of the LHC experiments"

Transcription

1 RAL, 13 May The ARDA project: Grid analysis prototypes of the LHC experiments Massimo Lamanna ARDA Project Leader cern.ch/lcg EGEE is a project funded by the European Union under contract IST

2 RAL, 13 May Contents ARDA Project Mandate and organisation ARDA activities during 2004 General pattern LHCb CMS ATLAS ALICE Conclusions and Outlook

3 ARDA working group recommendations: our starting point RAL, 13 May New service decomposition Strong influence of Alien system the Grid system developed by the ALICE experiments and used by a wide scientific community (not only HEP) Role of experience, existing technology Web service framework EGEE Middleware Interfacing to existing middleware to enable their use in the experiment frameworks Early deployment of (a series of) prototypes to ensure functionality and coherence ARDA project

4 RAL, 13 May EGEE and LCG Strong links already established between EDG and LCG. It will continue in the scope of EGEE The core infrastructure of the LCG and EGEE grids will be operated as a single service, and will grow out of LCG service LCG includes many US and Asia partners EGEE includes other sciences Substantial part of infrastructure common to both Parallel production lines as well LCG data challenges Pre production prototype EGEE MW ARDA playground for the LHC experiments ARDA LCG-1 LCG-2 EGEE-1 EGEE-2 VDT/EDG EGEE WS MW

5 RAL, 13 May End-to-end prototypes: why? Provide a fast feedback to the EGEE MW development team Avoid uncoordinated evolution of the middleware Coherence between users expectations and final product Experiments ready to benefit from the new MW as soon as possible Frequent snapshots of the middleware available Expose the experiments (and the community in charge of the deployment) to the current evolution of the whole system Experiments system are very complex and still evolving Move forward towards new-generation real systems (analysis!) Prototypes should be exercised with realistic workload and conditions No academic exercises or synthetic demonstrations LHC experiments users absolutely required here!!! EGEE Pilot Application A lot of work (experience and useful software) is involved in current experiments data challenges Concrete starting point Adapt/complete/refactorise the existing: we do not need another system!

6 RAL, 13 May End-to-end prototypes: how? The initial prototype will have a reduced scope Components selection for the first prototype Experiments components not in use for the first prototype are not ruled out (and used/selected ones might be replaced later on) Not all use cases/operation modes will be supported Every experiment has a production system (with multiple backends, like PBS, LCG, G2003, NorduGrid, ). We focus on end-user analysis on a EGEE MW based infrastructure Adapt/complete/refactorise the existing experiment (sub)system! Collaborative effort (not a parallel development) Attract and involve users Many users are absolutely required Informal Use Cases are still being defined, e.g.: A physicist selects a data sample (from current Data Challenges) With an example/template as starting point (s)he prepares a job to scan the data The job is split in sub-jobs, dispatched to the Grid, some error-recovery is automatically performed, merged back in a single output The output (histograms, ntuples) is returned together with simple information on the job-end status

7 RAL, 13 May Regional Centres Deployability is a key factor of MW success A few Regional Centres will have the responsibility to provide early installation for ARDA Understand Deployability issues Extend the ARDA test bed The ARDA test bed will be the next step after the most complex EGEE Middleware test bed Stress and performance tests could be ideally located outside CERN This is for experiment-specific components (e.g. a Meta Data catalogue) Leverage on Regional Centre local know how Data base technologies Web services Pilot sites might enlarge the resources available and give fundamental feedback in terms of deployability to complement the EGEE SA1 activity (EGEE/LCG operations) Running ARDA pilot installations Experiment data available where the experiment prototype is deployed

8 RAL, 13 May Coordination and forum activities The coordination activities would flow naturally from the fact that ARDA will be open to provide demonstration benches Since it is neither necessary nor possible that all projects could be hosted inside the ARDA experiments prototypes, some coordination is needed to ensure that new technologies can be exposed to the relevant community Transparent process ARDA should organise a set of regular meetings (one per quarter?) to discuss results, problems, new/alternative solutions and possibly agree on some coherent program of work. The ARDA project leader organises this activity which will be truly distributed and lead by the active partners ARDA is embedded in EGEE NA4 namely NA4-HEP Special relation with LCG GAG LCG forum for Grid requirements and use cases Experiments representatives coincide with the EGEE NA4 experiments representatives ARDA will channel this information to the appropriate recipients ARDA workshop (January 2004 at CERN; open; over 150 participants) ARDA workshop (June at CERN; by invitation) The first 30 days of EGEE middleware NA4 meeting mid July (NA4/JRA1 and NA4/SA1 sessions foreseen. Organised by M. Lamanna and F. Harris) ARDA workshop (September 2004?; open)

9 RAL, 13 May People Massimo Lamanna Birger Koblitz Andrey Demichev Viktor Pose Russia Dietrich Liko Frederik Orellana ALICE Wei-Long Ueng Tao-Sheng Chen Taiwan Derek Feichtinger Andreas Peters Julia Andreeva Juha Herrala Andrew Maier Kuba Moscicki ATLAS CMS LHCb Experiment interfaces Piergiorgio Cerello (ALICE) David Adams (ATLAS) Lucia Silvestris (CMS) Ulrik Egede (LHCb)

10 RAL, 13 May Example of activity Existing system as starting point Every experiment has different implementations of the standard services Used mainly in production environments Few expert users Coordinated update and read actions ARDA Interface with the EGEE middleware Verify (help to evolve to) such components to analysis environments Many users» Robustness Concurrent read actions» Performance One prototype per experiment A Common Application Layer might emerge in future ARDA emphasis is to enable each of the experiment to do its job Milestone Date Description Very very soon 1.x.1 May 2004 E2E x prototype definition agreed with the experiment 1.x.2 September 2004 E2E x prototype using basic EGEE middleware 1.x.3 November 2004 E2E x prototype improved functionality 1.x December 2004 E2E prototype for experiment x, capable of analysis Already started 2.x December 2005 E2E prototype for experiment x, capable of analysis and production

11 RAL, 13 May LHCb The LHCb system within ARDA uses GANGA as principal component (see next slide). The LHCb/GANGA plans: enable physicists (via GANGA) to analyse the data being produced during 2004 for their studies It naturally matches the ARDA mandate Have the prototype where the LHCb data will be the key At the beginning, the emphasis will be to validate the tool focusing on usability, validation of the splitting and merging functionality for users jobs The DIRAC system (LHCb grid system, used mainly in production so far, could be a useful playground to understand the detailed behaviour of some components, like the file catalog)

12 RAL, 13 May GANGA Gaudi/Athena and Grid Alliance Gaudi/Athena: LHCb/ATLAS frameworks The Athena uses Gaudi as a foundation Single desktop for a variety of tasks Help configuring and submitting analysis jobs Keep track of what they have done, hiding completely all technicalities Resource Broker, LSF, PBS, DIRAC, Condor Job registry stored locally or in the roaming profile Automate config/submit/monitor procedures Provide a palette of possible choices and specialized plug-ins (pre-defined application configurations, batch/grid systems, etc.) UI GUI JobOptions Algorithms BkSvc GANGA GAUDI Program Internal Model Histograms Monitoring Results Collective & Resource Grid Services WLM ProSvc Monitor GANGA Friendly user interface (CLI/GUI) is essential GUI Wizard Interface Help users to explore new capabilities Browse job registry Scripting/Command Line Interface Automate frequent tasks python shell embedded into the Ganga GUI Bookkeeping Service SE WorkLoad Manager CE File catalog Profile Service Grid Services Instr. GAUDI Program

13 RAL, 13 May ARDA contribution to Ganga Integration with EGEE middleware Waiting for the EGEE middleware, we developed an interface to Condor Use of Condor DAGMAN for splitting/merging and error recovery capability Design and Development Command Line Interface Future evolution of Ganga Release management Software process and integration Testing, tagging policies etc. Infrastructure Installation, packaging etc.

14 RAL, 13 May LHCb Metadata catalog Used in production (for large productions) Web Service layer being developed (main developers in the UK) Oracle backend ARDA contributes a testing focused on the analysis usage Robustness Performances under high concurrency (read mode) Measured network rate vs no. of concurrent clients

15 RAL, 13 May CERN/Taiwan tests Network monitor Virtual Users Client Oracle DB CERN Bookkeeping Server Web & XML-RPC Service performance tests CPU Load Network Process time Clone Bookkeeping DB in Taiwan Install the WS layer Performance Tests Database I/O Sensor Bookkeeping Server performance tests Taiwan/CERN Bookkeeping Server DB XML-RPC Service performance tests CPU Load, Network send/receive sensor, Process time Client Host performance tests CPU Load, Network send/receive sensor, Process time DB I/O Sensor Oracle DB CPU Load Network Process time Bookkeeping Server TAIWAN

16 RAL, 13 May CMS The CMS system within ARDA is still under discussion Provide easy access (and possibly sharing) of data for the CMS users is a key issue RefDB is the bookkeeping engine to plan and steer the production across different phases (simulation, reconstruction, to some degree into the analysis phase) It contained all necessary information except file physical location (RLS) and info related to the transfer management system (TMDB) The actual mechanism to provide these data to analysis users is under discussion Measuring performances underway (similar philosophy as for the LHCb Metadata catalog measurements) Reconstruction instructions RefDB McRunjob Tapes RefDB in CMS DC04 Reconstruction jobs Reconstructed data Summaries of successful jobs T0 worker nodes GDB castor pool Export Buffers Reconstructed data Checks what has arrived Updates RLS Transfer agent Updates TMDB

17 RAL, 13 May ATLAS The ATLAS system within ARDA has been agreed ATLAS has a complex strategy for distributed analysis, addressing different area with specific projects (Fast response, user-driven analysis, massive production, etc : see Starting point is the DIAL system The AMI metadata catalog is a key component mysql as a back end Genuine Web Server implementation Robustness and performance tests from ARDA In the start up phase, ARDA provided some help in developing ATLAS production tools Being finalised

18 RAL, 13 May What is DIAL? Interactive analysis e.g. ROOT, JAS,... DIAL Dataset Job Scheduler AAA Distributed processing running data-specific application

19 RAL, 13 May AMI studies in ARDA Atlas Metadata- Catalogue, contains File Metadata: Simulation/Reconstruction-Version Does not contain physical filenames Many problems still open: Large network traffic overhead due to schema independent tables SOAP proxy supposed to provide DB access Note that Web Services are stateless (not automatic handles to have the concept of session, transaction, etc ): 1 query = 1 (full) response Large queries might crashed server Shall proxy re-implement all database functionality? Good collaboration in place with ATLAS- Grenoble User User User SOAP-Proxy Meta-Data (MySQL) Studied behaviour using many concurrent clients:

20 ALICE: Grid enabled PROOF SuperComputing 2003 (SC2003) Demo RAL, 13 May PROOF SLAVES Site C Site A PROOF SLAVES TcpRouter Site B PROOF SLAVES Strategy: TcpRouter The ALICE/ARDA will evolve the analysis system presented by ALICE at SuperComputing 2003 With the new EGEE middleware (at SC2003, AliEn was used) Activity on PROOF Robustness Error recovery PROOF TcpRouter USER SESSION PROOF MASTER SERVER TcpRouter

21 RAL, 13 May ALICE-ARDA prototype improvements SC2003: The setup was heavily connected with the middleware services Somewhat inflexible configuration No chance to use PROOF on federated grids like LCG in AliEn TcpRouter service needs incoming connectivity in each site Libraries can not be distributed using the standard rootd functionality Improvement ideas: Distribute another daemon with ROOT, which replaces the need for a TcpRouter service Connect each slave proofd/rootd via this daemon to two central proofd/rootd master multiplexer daemons, which run together with the proof master Use Grid functionality for daemon start-up and booking policies through a plug-in interface from ROOT Put PROOF/ROOT on top of the grid services Improve on dynamic configuration and error recovery

22 RAL, 13 May ALICE-ARDA improved system PROOF SLAVE SERVERS Proxy proofd Proxy rootd Grid Services Booking The remote proof slaves look like a local proof slave on the master machine Booking service is usable also on local clusters PROOF Master

23 RAL, 13 May Conclusions and Outlook ARDA is starting Main tool: experiment prototypes for analysis Detailed project plan being prepared Good feedback from the LHC experiments Good collaboration with EGEE NA4 Good collaboration with Regional Centres. More help needed Look forward to contribute to the success of EGEE Helping EGEE Middleware to deliver a fully functionally solution ARDA main focus Collaborate with the LHC experiments to set up the end-to-end prototypes Aggressive schedule First milestone for the end-to-end prototypes is Dec 2004

ARDA status. Massimo Lamanna / CERN. LHCC referees meeting, 28 June

ARDA status. Massimo Lamanna / CERN. LHCC referees meeting, 28 June LHCC referees meeting, 28 June 2004 ARDA status http://cern.ch/arda Massimo Lamanna / CERN www.eu-egee.org cern.ch/lcg EGEE is a project funded by the European Union under contract IST-2003-508833 LHCC

More information

AMGA metadata catalogue system

AMGA metadata catalogue system AMGA metadata catalogue system Hurng-Chun Lee ACGrid School, Hanoi, Vietnam www.eu-egee.org EGEE and glite are registered trademarks Outline AMGA overview AMGA Background and Motivation for AMGA Interface,

More information

The LHC Computing Grid

The LHC Computing Grid The LHC Computing Grid Gergely Debreczeni (CERN IT/Grid Deployment Group) The data factory of LHC 40 million collisions in each second After on-line triggers and selections, only 100 3-4 MB/event requires

More information

where the Web was born Experience of Adding New Architectures to the LCG Production Environment

where the Web was born Experience of Adding New Architectures to the LCG Production Environment where the Web was born Experience of Adding New Architectures to the LCG Production Environment Andreas Unterkircher, openlab fellow Sverre Jarp, CTO CERN openlab Industrializing the Grid openlab Workshop

More information

Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science

Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science T. Maeno, K. De, A. Klimentov, P. Nilsson, D. Oleynik, S. Panitkin, A. Petrosyan, J. Schovancova, A. Vaniachine,

More information

Batch Services at CERN: Status and Future Evolution

Batch Services at CERN: Status and Future Evolution Batch Services at CERN: Status and Future Evolution Helge Meinhard, CERN-IT Platform and Engineering Services Group Leader HTCondor Week 20 May 2015 20-May-2015 CERN batch status and evolution - Helge

More information

Workload Management. Stefano Lacaprara. CMS Physics Week, FNAL, 12/16 April Department of Physics INFN and University of Padova

Workload Management. Stefano Lacaprara. CMS Physics Week, FNAL, 12/16 April Department of Physics INFN and University of Padova Workload Management Stefano Lacaprara Department of Physics INFN and University of Padova CMS Physics Week, FNAL, 12/16 April 2005 Outline 1 Workload Management: the CMS way General Architecture Present

More information

Summary of the LHC Computing Review

Summary of the LHC Computing Review Summary of the LHC Computing Review http://lhc-computing-review-public.web.cern.ch John Harvey CERN/EP May 10 th, 2001 LHCb Collaboration Meeting The Scale Data taking rate : 50,100, 200 Hz (ALICE, ATLAS-CMS,

More information

ELFms industrialisation plans

ELFms industrialisation plans ELFms industrialisation plans CERN openlab workshop 13 June 2005 German Cancio CERN IT/FIO http://cern.ch/elfms ELFms industrialisation plans, 13/6/05 Outline Background What is ELFms Collaboration with

More information

CHIPP Phoenix Cluster Inauguration

CHIPP Phoenix Cluster Inauguration TheComputing Environment for LHC Data Analysis The LHC Computing Grid CHIPP Phoenix Cluster Inauguration Manno, Switzerland 30 May 2008 Les Robertson IT Department - CERN CH-1211 Genève 23 les.robertson@cern.ch

More information

The CORAL Project. Dirk Düllmann for the CORAL team Open Grid Forum, Database Workshop Barcelona, 4 June 2008

The CORAL Project. Dirk Düllmann for the CORAL team Open Grid Forum, Database Workshop Barcelona, 4 June 2008 The CORAL Project Dirk Düllmann for the CORAL team Open Grid Forum, Database Workshop Barcelona, 4 June 2008 Outline CORAL - a foundation for Physics Database Applications in the LHC Computing Grid (LCG)

More information

EGEE and Interoperation

EGEE and Interoperation EGEE and Interoperation Laurence Field CERN-IT-GD ISGC 2008 www.eu-egee.org EGEE and glite are registered trademarks Overview The grid problem definition GLite and EGEE The interoperability problem The

More information

Computing for LHC in Germany

Computing for LHC in Germany 1 Computing for LHC in Germany Günter Quast Universität Karlsruhe (TH) Meeting with RECFA Berlin, October 5th 2007 WLCG Tier1 & Tier2 Additional resources for data analysis - HGF ''Physics at the Terascale''

More information

The grid for LHC Data Analysis

The grid for LHC Data Analysis The grid for LHC Data Analysis ICAP 2006 Conference Chamonix 5 October 2006 Les Robertson - CERN LHC Computing Grid Project Leader The LHC Computing Challenges 1. Data After reduction by triggers and data

More information

GRIDS INTRODUCTION TO GRID INFRASTRUCTURES. Fabrizio Gagliardi

GRIDS INTRODUCTION TO GRID INFRASTRUCTURES. Fabrizio Gagliardi GRIDS INTRODUCTION TO GRID INFRASTRUCTURES Fabrizio Gagliardi Dr. Fabrizio Gagliardi is the leader of the EU DataGrid project and designated director of the proposed EGEE (Enabling Grids for E-science

More information

The LHC Computing Grid

The LHC Computing Grid The LHC Computing Grid Visit of Finnish IT Centre for Science CSC Board Members Finland Tuesday 19 th May 2009 Frédéric Hemmer IT Department Head The LHC and Detectors Outline Computing Challenges Current

More information

Interoperating AliEn and ARC for a distributed Tier1 in the Nordic countries.

Interoperating AliEn and ARC for a distributed Tier1 in the Nordic countries. for a distributed Tier1 in the Nordic countries. Philippe Gros Lund University, Div. of Experimental High Energy Physics, Box 118, 22100 Lund, Sweden philippe.gros@hep.lu.se Anders Rhod Gregersen NDGF

More information

Analisi Tier2 e Tier3 Esperienze ai Tier-2 Giacinto Donvito INFN-BARI

Analisi Tier2 e Tier3 Esperienze ai Tier-2 Giacinto Donvito INFN-BARI Analisi Tier2 e Tier3 Esperienze ai Tier-2 Giacinto Donvito INFN-BARI outlook Alice Examples Atlas Examples CMS Examples Alice Examples ALICE Tier-2s at the moment do not support interactive analysis not

More information

LCG Conditions Database Project

LCG Conditions Database Project Computing in High Energy and Nuclear Physics (CHEP 2006) TIFR, Mumbai, 13 Feb 2006 LCG Conditions Database Project COOL Development and Deployment: Status and Plans On behalf of the COOL team (A.V., D.Front,

More information

ALICE Grid/Analysis Tutorial Exercise-Solutions

ALICE Grid/Analysis Tutorial Exercise-Solutions WLCG Asia Workshop, 2.12.2006 ALICE Grid/Analysis Tutorial Exercise-Solutions Andreas-Joachim Peters CERN www.eu-egee.org cern.ch/lcg http://cern.ch/arda EGEE is a project funded by the European Union

More information

Streamlining CASTOR to manage the LHC data torrent

Streamlining CASTOR to manage the LHC data torrent Streamlining CASTOR to manage the LHC data torrent G. Lo Presti, X. Espinal Curull, E. Cano, B. Fiorini, A. Ieri, S. Murray, S. Ponce and E. Sindrilaru CERN, 1211 Geneva 23, Switzerland E-mail: giuseppe.lopresti@cern.ch

More information

Data Management for the World s Largest Machine

Data Management for the World s Largest Machine Data Management for the World s Largest Machine Sigve Haug 1, Farid Ould-Saada 2, Katarina Pajchel 2, and Alexander L. Read 2 1 Laboratory for High Energy Physics, University of Bern, Sidlerstrasse 5,

More information

WHEN the Large Hadron Collider (LHC) begins operation

WHEN the Large Hadron Collider (LHC) begins operation 2228 IEEE TRANSACTIONS ON NUCLEAR SCIENCE, VOL. 53, NO. 4, AUGUST 2006 Measurement of the LCG2 and Glite File Catalogue s Performance Craig Munro, Birger Koblitz, Nuno Santos, and Akram Khan Abstract When

More information

Framework for Interactive Parallel Dataset Analysis on the Grid

Framework for Interactive Parallel Dataset Analysis on the Grid SLAC-PUB-12289 January 2007 Framework for Interactive Parallel Analysis on the David A. Alexander, Balamurali Ananthan Tech-X Corporation 5621 Arapahoe Ave, Suite A Boulder, CO 80303 {alexanda,bala}@txcorp.com

More information

ISTITUTO NAZIONALE DI FISICA NUCLEARE

ISTITUTO NAZIONALE DI FISICA NUCLEARE ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Perugia INFN/TC-05/10 July 4, 2005 DESIGN, IMPLEMENTATION AND CONFIGURATION OF A GRID SITE WITH A PRIVATE NETWORK ARCHITECTURE Leonello Servoli 1,2!, Mirko

More information

Grids and Security. Ian Neilson Grid Deployment Group CERN. TF-CSIRT London 27 Jan

Grids and Security. Ian Neilson Grid Deployment Group CERN. TF-CSIRT London 27 Jan Grids and Security Ian Neilson Grid Deployment Group CERN TF-CSIRT London 27 Jan 2004-1 TOC Background Grids Grid Projects Some Technical Aspects The three or four A s Some Operational Aspects Security

More information

The LHC Computing Grid. Slides mostly by: Dr Ian Bird LCG Project Leader 18 March 2008

The LHC Computing Grid. Slides mostly by: Dr Ian Bird LCG Project Leader 18 March 2008 The LHC Computing Grid Slides mostly by: Dr Ian Bird LCG Project Leader 18 March 2008 The LHC Computing Grid February 2008 Some precursors Computing for HEP means data handling Fixed-target experiments

More information

The Grid: Processing the Data from the World s Largest Scientific Machine

The Grid: Processing the Data from the World s Largest Scientific Machine The Grid: Processing the Data from the World s Largest Scientific Machine 10th Topical Seminar On Innovative Particle and Radiation Detectors Siena, 1-5 October 2006 Patricia Méndez Lorenzo (IT-PSS/ED),

More information

The LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland

The LCG 3D Project. Maria Girone, CERN. The 23rd Open Grid Forum - OGF23 4th June 2008, Barcelona. CERN IT Department CH-1211 Genève 23 Switzerland The LCG 3D Project Maria Girone, CERN The rd Open Grid Forum - OGF 4th June 2008, Barcelona Outline Introduction The Distributed Database (3D) Project Streams Replication Technology and Performance Availability

More information

Cooperation among ALICE Storage Elements: current status and directions (The ALICE Global Redirector: a step towards real storage robustness).

Cooperation among ALICE Storage Elements: current status and directions (The ALICE Global Redirector: a step towards real storage robustness). Cooperation among ALICE Storage Elements: current status and directions (The ALICE Global Redirector: a step towards real storage robustness). 1 CERN Geneve 23, CH-1211, Switzerland E-mail: fabrizio.furano@cern.ch

More information

Challenges of the LHC Computing Grid by the CMS experiment

Challenges of the LHC Computing Grid by the CMS experiment 2007 German e-science Available online at http://www.ges2007.de This document is under the terms of the CC-BY-NC-ND Creative Commons Attribution Challenges of the LHC Computing Grid by the CMS experiment

More information

Evolution of Database Replication Technologies for WLCG

Evolution of Database Replication Technologies for WLCG Journal of Physics: Conference Series PAPER OPEN ACCESS Evolution of Database Replication Technologies for WLCG To cite this article: Zbigniew Baranowski et al 2015 J. Phys.: Conf. Ser. 664 042032 View

More information

13th International Workshop on Advanced Computing and Analysis Techniques in Physics Research ACAT 2010 Jaipur, India February

13th International Workshop on Advanced Computing and Analysis Techniques in Physics Research ACAT 2010 Jaipur, India February LHC Cloud Computing with CernVM Ben Segal 1 CERN 1211 Geneva 23, Switzerland E mail: b.segal@cern.ch Predrag Buncic CERN E mail: predrag.buncic@cern.ch 13th International Workshop on Advanced Computing

More information

Operating the Distributed NDGF Tier-1

Operating the Distributed NDGF Tier-1 Operating the Distributed NDGF Tier-1 Michael Grønager Technical Coordinator, NDGF International Symposium on Grid Computing 08 Taipei, April 10th 2008 Talk Outline What is NDGF? Why a distributed Tier-1?

More information

High Throughput WAN Data Transfer with Hadoop-based Storage

High Throughput WAN Data Transfer with Hadoop-based Storage High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San

More information

IEPSAS-Kosice: experiences in running LCG site

IEPSAS-Kosice: experiences in running LCG site IEPSAS-Kosice: experiences in running LCG site Marian Babik 1, Dusan Bruncko 2, Tomas Daranyi 1, Ladislav Hluchy 1 and Pavol Strizenec 2 1 Department of Parallel and Distributed Computing, Institute of

More information

Data Analysis in ATLAS. Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser

Data Analysis in ATLAS. Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser Data Analysis in ATLAS Graeme Stewart with thanks to Attila Krasznahorkay and Johannes Elmsheuser 1 ATLAS Data Flow into Analysis RAW detector data and simulated RDO data are reconstructed into our xaod

More information

FREE SCIENTIFIC COMPUTING

FREE SCIENTIFIC COMPUTING Institute of Physics, Belgrade Scientific Computing Laboratory FREE SCIENTIFIC COMPUTING GRID COMPUTING Branimir Acković March 4, 2007 Petnica Science Center Overview 1/2 escience Brief History of UNIX

More information

THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap

THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap Arnie Miles Georgetown University adm35@georgetown.edu http://thebes.arc.georgetown.edu The Thebes middleware project was

More information

Recent developments in user-job management with Ganga

Recent developments in user-job management with Ganga Recent developments in user-job management with Ganga Currie R 1, Elmsheuser J 2, Fay R 3, Owen P H 1, Richards A 1, Slater M 4, Sutcliffe W 1, Williams M 4 1 Blackett Laboratory, Imperial College London,

More information

Evolution of Database Replication Technologies for WLCG

Evolution of Database Replication Technologies for WLCG Evolution of Database Replication Technologies for WLCG Zbigniew Baranowski, Lorena Lobato Pardavila, Marcin Blaszczyk, Gancho Dimitrov, Luca Canali European Organisation for Nuclear Research (CERN), CH-1211

More information

First Experience with LCG. Board of Sponsors 3 rd April 2009

First Experience with LCG. Board of Sponsors 3 rd April 2009 First Experience with LCG Operation and the future... CERN openlab Board of Sponsors 3 rd April 2009 Ian Bird LCG Project Leader The LHC Computing Challenge Signal/Noise: 10-9 Data volume High rate * large

More information

Overview of ATLAS PanDA Workload Management

Overview of ATLAS PanDA Workload Management Overview of ATLAS PanDA Workload Management T. Maeno 1, K. De 2, T. Wenaus 1, P. Nilsson 2, G. A. Stewart 3, R. Walker 4, A. Stradling 2, J. Caballero 1, M. Potekhin 1, D. Smith 5, for The ATLAS Collaboration

More information

Data Access and Data Management

Data Access and Data Management Data Access and Data Management in grids Jos van Wezel Overview Background [KIT, GridKa] Practice [LHC, glite] Data storage systems [dcache a.o.] Data and meta data Intro KIT = FZK + Univ. of Karlsruhe

More information

Computing grids, a tool for international collaboration and against digital divide Guy Wormser Director of CNRS Institut des Grilles (CNRS, France)

Computing grids, a tool for international collaboration and against digital divide Guy Wormser Director of CNRS Institut des Grilles (CNRS, France) Computing grids, a tool for international collaboration and against digital divide Guy Wormser Director of CNRS Institut des Grilles (CNRS, France) www.eu-egee.org EGEE and glite are registered trademarks

More information

Federated data storage system prototype for LHC experiments and data intensive science

Federated data storage system prototype for LHC experiments and data intensive science Federated data storage system prototype for LHC experiments and data intensive science A. Kiryanov 1,2,a, A. Klimentov 1,3,b, D. Krasnopevtsev 1,4,c, E. Ryabinkin 1,d, A. Zarochentsev 1,5,e 1 National

More information

Application of Virtualization Technologies & CernVM. Benedikt Hegner CERN

Application of Virtualization Technologies & CernVM. Benedikt Hegner CERN Application of Virtualization Technologies & CernVM Benedikt Hegner CERN Virtualization Use Cases Worker Node Virtualization Software Testing Training Platform Software Deployment }Covered today Server

More information

RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP

RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP THE EUROPEAN ORGANISATION FOR PARTICLE PHYSICS RESEARCH (CERN) 2 THE LARGE HADRON COLLIDER THE LARGE HADRON COLLIDER

More information

The glite middleware. Ariel Garcia KIT

The glite middleware. Ariel Garcia KIT The glite middleware Ariel Garcia KIT Overview Background The glite subsystems overview Security Information system Job management Data management Some (my) answers to your questions and random rumblings

More information

CouchDB-based system for data management in a Grid environment Implementation and Experience

CouchDB-based system for data management in a Grid environment Implementation and Experience CouchDB-based system for data management in a Grid environment Implementation and Experience Hassen Riahi IT/SDC, CERN Outline Context Problematic and strategy System architecture Integration and deployment

More information

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF Conference 2017 The Data Challenges of the LHC Reda Tafirout, TRIUMF Outline LHC Science goals, tools and data Worldwide LHC Computing Grid Collaboration & Scale Key challenges Networking ATLAS experiment

More information

AGIS: The ATLAS Grid Information System

AGIS: The ATLAS Grid Information System AGIS: The ATLAS Grid Information System Alexey Anisenkov 1, Sergey Belov 2, Alessandro Di Girolamo 3, Stavro Gayazov 1, Alexei Klimentov 4, Danila Oleynik 2, Alexander Senchenko 1 on behalf of the ATLAS

More information

Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing

Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing Wolf Behrenhoff, Christoph Wissing DESY Computing Seminar May 17th, 2010 Page 1 Installation of

More information

Monitoring the Usage of the ZEUS Analysis Grid

Monitoring the Usage of the ZEUS Analysis Grid Monitoring the Usage of the ZEUS Analysis Grid Stefanos Leontsinis September 9, 2006 Summer Student Programme 2006 DESY Hamburg Supervisor Dr. Hartmut Stadie National Technical

More information

Multiple Broker Support by Grid Portals* Extended Abstract

Multiple Broker Support by Grid Portals* Extended Abstract 1. Introduction Multiple Broker Support by Grid Portals* Extended Abstract Attila Kertesz 1,3, Zoltan Farkas 1,4, Peter Kacsuk 1,4, Tamas Kiss 2,4 1 MTA SZTAKI Computer and Automation Research Institute

More information

A European Vision and Plan for a Common Grid Infrastructure

A European Vision and Plan for a Common Grid Infrastructure A European Vision and Plan for a Common Grid Infrastructure European Grid Initiative www.eu-egi.org Why Sustainability? Scientific applications start to depend on Grid infrastructures (EGEE, DEISA, ) Jobs/month

More information

Grid Computing Activities at KIT

Grid Computing Activities at KIT Grid Computing Activities at KIT Meeting between NCP and KIT, 21.09.2015 Manuel Giffels Karlsruhe Institute of Technology Institute of Experimental Nuclear Physics & Steinbuch Center for Computing Courtesy

More information

Expressing Parallelism with ROOT

Expressing Parallelism with ROOT Expressing Parallelism with ROOT https://root.cern D. Piparo (CERN) for the ROOT team CHEP 2016 2 This Talk ROOT helps scientists to express parallelism Adopting multi-threading (MT) and multi-processing

More information

Volunteer Computing at CERN

Volunteer Computing at CERN Volunteer Computing at CERN BOINC workshop Sep 2014, Budapest Tomi Asp & Pete Jones, on behalf the LHC@Home team Agenda Overview Status of the LHC@Home projects Additional BOINC projects Service consolidation

More information

DIONE. (DAML Integrated Ontology Evolution Tools) Ontology Versioning in Semantic Web Applications. ISX Corporation Lehigh University

DIONE. (DAML Integrated Ontology Evolution Tools) Ontology Versioning in Semantic Web Applications. ISX Corporation Lehigh University (DAML Integrated Evolution Tools) Versioning in Semantic Web Applications ISX Corporation Lehigh University Dr. Brian Kettler, ISX bkettler@isx.com Prof. Jeff Heflin & Zhengxiang Pan, Lehigh heflin@cse.lehigh.edu

More information

Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland

Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Online data storage service strategy for the CERN computer Centre G. Cancio, D. Duellmann, M. Lamanna, A. Pace CERN, Geneva, Switzerland Abstract. The Data and Storage Services group at CERN is conducting

More information

Agent-Enabling Transformation of E-Commerce Portals with Web Services

Agent-Enabling Transformation of E-Commerce Portals with Web Services Agent-Enabling Transformation of E-Commerce Portals with Web Services Dr. David B. Ulmer CTO Sotheby s New York, NY 10021, USA Dr. Lixin Tao Professor Pace University Pleasantville, NY 10570, USA Abstract:

More information

Garuda : The National Grid Computing Initiative Of India. Natraj A.C, CDAC Knowledge Park, Bangalore.

Garuda : The National Grid Computing Initiative Of India. Natraj A.C, CDAC Knowledge Park, Bangalore. Garuda : The National Grid Computing Initiative Of India Natraj A.C, CDAC Knowledge Park, Bangalore. natraj@cdacb.ernet.in 1 Agenda About CDAC Garuda grid highlights Garuda Foundation Phase EU-India grid

More information

Existing Tools in HEP and Particle Astrophysics

Existing Tools in HEP and Particle Astrophysics Existing Tools in HEP and Particle Astrophysics Richard Dubois richard@slac.stanford.edu R.Dubois Existing Tools in HEP and Particle Astro 1/20 Outline Introduction: Fermi as example user Analysis Toolkits:

More information

Physics Computing at CERN. Helge Meinhard CERN, IT Department OpenLab Student Lecture 27 July 2010

Physics Computing at CERN. Helge Meinhard CERN, IT Department OpenLab Student Lecture 27 July 2010 Physics Computing at CERN Helge Meinhard CERN, IT Department OpenLab Student Lecture 27 July 2010 Location Building 513 (opposite of restaurant no. 2) Building Large building with 2700 m 2 surface for

More information

Lessons Learned in the NorduGrid Federation

Lessons Learned in the NorduGrid Federation Lessons Learned in the NorduGrid Federation David Cameron University of Oslo With input from Gerd Behrmann, Oxana Smirnova and Mattias Wadenstein Creating Federated Data Stores For The LHC 14.9.12, Lyon,

More information

DIRAC Distributed Infrastructure with Remote Agent Control

DIRAC Distributed Infrastructure with Remote Agent Control Computing in High Energy and Nuclear Physics, La Jolla, California, 24-28 March 2003 1 DIRAC Distributed Infrastructure with Remote Agent Control A.Tsaregorodtsev, V.Garonne CPPM-IN2P3-CNRS, Marseille,

More information

CERN openlab II. CERN openlab and. Sverre Jarp CERN openlab CTO 16 September 2008

CERN openlab II. CERN openlab and. Sverre Jarp CERN openlab CTO 16 September 2008 CERN openlab II CERN openlab and Intel: Today and Tomorrow Sverre Jarp CERN openlab CTO 16 September 2008 Overview of CERN 2 CERN is the world's largest particle physics centre What is CERN? Particle physics

More information

Grid Computing a new tool for science

Grid Computing a new tool for science Grid Computing a new tool for science CERN, the European Organization for Nuclear Research Dr. Wolfgang von Rüden Wolfgang von Rüden, CERN, IT Department Grid Computing July 2006 CERN stands for over 50

More information

Grid Compute Resources and Job Management

Grid Compute Resources and Job Management Grid Compute Resources and Job Management How do we access the grid? Command line with tools that you'll use Specialised applications Ex: Write a program to process images that sends data to run on the

More information

CernVM-FS beyond LHC computing

CernVM-FS beyond LHC computing CernVM-FS beyond LHC computing C Condurache, I Collier STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot, OX11 0QX, UK E-mail: catalin.condurache@stfc.ac.uk Abstract. In the last three years

More information

The AliEn system, status and perspectives

The AliEn system, status and perspectives CHEP03, March 24-28, 2003, La Jolla, California, U.S.A. 1 The AliEn system, status and perspectives P. Buncic Institut für Kernphysik, August-Euler-Str. 6, D-60486 Frankfurt, Germany and CERN, 1211, Geneva

More information

Towards Network Awareness in LHC Computing

Towards Network Awareness in LHC Computing Towards Network Awareness in LHC Computing CMS ALICE CERN Atlas LHCb LHC Run1: Discovery of a New Boson LHC Run2: Beyond the Standard Model Gateway to a New Era Artur Barczyk / Caltech Internet2 Technology

More information

Pan-European Grid einfrastructure for LHC Experiments at CERN - SCL's Activities in EGEE

Pan-European Grid einfrastructure for LHC Experiments at CERN - SCL's Activities in EGEE Pan-European Grid einfrastructure for LHC Experiments at CERN - SCL's Activities in EGEE Aleksandar Belić Scientific Computing Laboratory Institute of Physics EGEE Introduction EGEE = Enabling Grids for

More information

The ATLAS Production System

The ATLAS Production System The ATLAS MC and Data Rodney Walker Ludwig Maximilians Universität Munich 2nd Feb, 2009 / DESY Computing Seminar Outline 1 Monte Carlo Production Data 2 3 MC Production Data MC Production Data Group and

More information

The ATLAS METADATA INTERFACE

The ATLAS METADATA INTERFACE Author manuscript, published in "International Conference on Computing in High Energy and Nuclear Physics (CHEP-07), Victoria : Canada (2007)" DOI : 10.1088/1742-6596/120/7/072003 The ATLAS METADATA INTERFACE

More information

A Prototype of the CMS Object Oriented Reconstruction and Analysis Framework for the Beam Test Data

A Prototype of the CMS Object Oriented Reconstruction and Analysis Framework for the Beam Test Data Prototype of the CMS Object Oriented Reconstruction and nalysis Framework for the Beam Test Data CMS Collaboration presented by Lucia Silvestris CERN, Geneve, Suisse and INFN, Bari, Italy bstract. CMS

More information

Chapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT.

Chapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT. Chapter 4:- Introduction to Grid and its Evolution Prepared By:- Assistant Professor SVBIT. Overview Background: What is the Grid? Related technologies Grid applications Communities Grid Tools Case Studies

More information

Data Transfers Between LHC Grid Sites Dorian Kcira

Data Transfers Between LHC Grid Sites Dorian Kcira Data Transfers Between LHC Grid Sites Dorian Kcira dkcira@caltech.edu Caltech High Energy Physics Group hep.caltech.edu/cms CERN Site: LHC and the Experiments Large Hadron Collider 27 km circumference

More information

LCG Installation LCFGng

LCG Installation LCFGng GridKa School, 20-23 September 2004 www.eu-egee.org LCG Installation LCFGng Peer Hasselmeyer GridKa, FZK EGEE is a project funded by the European Union under contract IST-2003-508833 What is LCFGng? LCFGng

More information

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of

More information

Oracle and Tangosol Acquisition Announcement

Oracle and Tangosol Acquisition Announcement Oracle and Tangosol Acquisition Announcement March 23, 2007 The following is intended to outline our general product direction. It is intended for information purposes only, and may

More information

Physics Computing at CERN. Helge Meinhard CERN, IT Department OpenLab Student Lecture 21 July 2011

Physics Computing at CERN. Helge Meinhard CERN, IT Department OpenLab Student Lecture 21 July 2011 Physics Computing at CERN Helge Meinhard CERN, IT Department OpenLab Student Lecture 21 July 2011 Location (1) Building 513 (opposite of restaurant no. 2) Building 513 (1) Large building with 2700 m 2

More information

Microsoft Office Communicator 2007 R2 Getting Started Guide. Published: December 2008

Microsoft Office Communicator 2007 R2 Getting Started Guide. Published: December 2008 Microsoft Office Communicator 2007 R2 Getting Started Guide Published: December 2008 Information in this document, including URL and other Internet Web site references, is subject to change without notice.

More information

LCG User Registration & VO management

LCG User Registration & VO management LCG User Registration & VO management Spring HEPiX Edinburgh 1Maria Dimou- cern-it-gd Presentation Outline Why is LCG Registration worth talking about. How do we register users today. What needs to be

More information

DIRECTORS OF METHODOLOGY/IT DIRECTORS JOINT STEERING GROUP 18 NOVEMBER 2015

DIRECTORS OF METHODOLOGY/IT DIRECTORS JOINT STEERING GROUP 18 NOVEMBER 2015 DIME/ITDG SG November 2015 DIRECTORS OF METHODOLOGY/IT DIRECTORS JOINT STEERING GROUP 18 NOVEMBER 2015 Item 03 of the agenda ESS Vision 2020: ESS.VIP VALIDATION 1. Purpose of the document ESS Vision 2020:

More information

BOSS and LHC computing using CernVM and BOINC

BOSS and LHC computing using CernVM and BOINC BOSS and LHC computing using CernVM and BOINC otn-2010-0x openlab Summer Student Report BOSS and LHC computing using CernVM and BOINC Jie Wu (Supervisor: Ben Segal / IT) 1 December 2010 Version 1 Distribution::

More information

PoS(ACAT)020. Status and evolution of CRAB. Fabio Farina University and INFN Milano-Bicocca S. Lacaprara INFN Legnaro

PoS(ACAT)020. Status and evolution of CRAB. Fabio Farina University and INFN Milano-Bicocca   S. Lacaprara INFN Legnaro Status and evolution of CRAB University and INFN Milano-Bicocca E-mail: fabio.farina@cern.ch S. Lacaprara INFN Legnaro W. Bacchi University and INFN Bologna M. Cinquilli University and INFN Perugia G.

More information

Prototype DIRAC portal for EISCAT data Short instruction

Prototype DIRAC portal for EISCAT data Short instruction Prototype DIRAC portal for EISCAT data Short instruction Carl-Fredrik Enell January 19, 2017 1 Introduction 1.1 DIRAC EGI, first European Grid Initiative, later European Grid Infrastructure, and now simply

More information

Global Software Distribution with CernVM-FS

Global Software Distribution with CernVM-FS Global Software Distribution with CernVM-FS Jakob Blomer CERN 2016 CCL Workshop on Scalable Computing October 19th, 2016 jblomer@cern.ch CernVM-FS 1 / 15 The Anatomy of a Scientific Software Stack (In

More information

Features and Future. Frédéric Hemmer - CERN Deputy Head of IT Department. Enabling Grids for E-sciencE. BEGrid seminar Brussels, October 27, 2006

Features and Future. Frédéric Hemmer - CERN Deputy Head of IT Department. Enabling Grids for E-sciencE. BEGrid seminar Brussels, October 27, 2006 Features and Future Frédéric Hemmer - CERN Deputy Head of IT Department BEGrid seminar Brussels, October 27, 2006 www.eu-egee.org www.glite.org Outline Overview of EGEE EGEE glite Middleware Foundation

More information

FilmGrid: Digital Asset Management for Film PostProduction

FilmGrid: Digital Asset Management for Film PostProduction FilmGrid: Digital Asset Management for Film PostProduction Wednesday 4 June 2008 Charaka Palansuriya EPCC, The University of Edinburgh charaka@epcc.ed.ac.uk Outline 2 What is FilmGrid Background to Film

More information

Helix Nebula The Science Cloud

Helix Nebula The Science Cloud Helix Nebula The Science Cloud Deliverable Title: Data Management Plan Partner Responsible: CERN Work Package: 1 Submission Due Date: 26 January 2016 Actual Submission Date: 23 March 2016 Distribution:

More information

Deploying virtualisation in a production grid

Deploying virtualisation in a production grid Deploying virtualisation in a production grid Stephen Childs Trinity College Dublin & Grid-Ireland TERENA NRENs and Grids workshop 2 nd September 2008 www.eu-egee.org EGEE and glite are registered trademarks

More information

Constant monitoring of multi-site network connectivity at the Tokyo Tier2 center

Constant monitoring of multi-site network connectivity at the Tokyo Tier2 center Constant monitoring of multi-site network connectivity at the Tokyo Tier2 center, T. Mashimo, N. Matsui, H. Matsunaga, H. Sakamoto, I. Ueda International Center for Elementary Particle Physics, The University

More information

WP JRA1: Architectures for an integrated and interoperable AAI

WP JRA1: Architectures for an integrated and interoperable AAI Authentication and Authorisation for Research and Collaboration WP JRA1: Architectures for an integrated and interoperable AAI Christos Kanellopoulos Agenda Structure and administrative matters Objectives

More information

Grid Services Security Vulnerability and Risk Analysis

Grid Services Security Vulnerability and Risk Analysis Grid Services Security Vulnerability and Risk Analysis Dr Linda Cornwall RAL www.eu-egee.org EGEE and glite are registered trademarks Contents Why we setup the Grid Security Vulnerability Group Starting

More information

Digital (Virtual) Identities in Daidalos and beyond. Amardeo Sarma NEC Laboratories Europe

Digital (Virtual) Identities in Daidalos and beyond. Amardeo Sarma NEC Laboratories Europe Digital (Virtual) Identities in Daidalos and beyond Amardeo Sarma NEC Laboratories Europe Who wants to pay for more Bandwidth? More Access Bandwidth? No one pays extra for volume or time plain usage is

More information

DELOS WP7: Evaluation

DELOS WP7: Evaluation DELOS WP7: Evaluation Claus-Peter Klas Univ. of Duisburg-Essen, Germany (WP leader: Norbert Fuhr) WP Objectives Enable communication between evaluation experts and DL researchers/developers Continue existing

More information

IllustraCve Example of Distributed Analysis in ATLAS Spanish Tier2 and Tier3

IllustraCve Example of Distributed Analysis in ATLAS Spanish Tier2 and Tier3 IllustraCve Example of Distributed Analysis in ATLAS Spanish Tier2 and Tier3 S. González, E. Oliver, M. Villaplana, A. Fernández, M. Kaci, A. Lamas, J. Salt, J. Sánchez PCI2010 Workshop Rabat, 5 th 7 th

More information