CRAB 2 and CRABServer at UCSD

Size: px
Start display at page:

Download "CRAB 2 and CRABServer at UCSD"

Transcription

1 CRAB 2 and CRABServer at UCSD JAMES LETTS* (with advice and slides from Sanjay Padhi) - UCSD - * I have never been a CRAB developer or even been responsible for running a CRABServer, so if you see errors in this talk, please jump in and correct me... 1

2 Basics UCSD hosts one of the CRAB Servers for CMS Analysis with CRAB 2 Other servers at CERN, Bari, Desy... From the glidein point-of-view, the CRAB Server is a scheduler (schedd) These slides present some of the features and internal workings of CRAB Server 2

3 Caveat CRAB 2 is effectively legacy software Will be replaced by CRAB 3 (WMAgent) CRAB 2 will be maintained by CMS Analysis Operations* until at least November No significant development effort Emphasis on stability and maintaining the present functionality 3

4 CRAB 2 Modes Stand-alone CRAB, and CRAB Server glite and glidein submitters Users can submit tasks in any combination by appropriate crab.cfg: #scheduler=glite scheduler=glidein use_server=1 4

5 Stand-alone vs. Server The standalone model has the advantage of simplicity and stability... But lacks features provided by a more advanced client-server architecture. CRAB Server is placed between the user and the Grid to perform a set of actions. Automate more the analysis work flow and to improve the scalability of the system. 5

6 Features of CRAB Server Job monitoring with a web-based front end Automatic resubmission of aborted jobs Notification of job progress User support from Analysis Operations hn-cms-crabfeedback@cern.ch General principle to isolate user as much as possible from the Grid: submission, job/task monitoring, output retrieval. 6

7 CRAB 2 Server CRAB Server tries to isolate the user from the Grid User interface is same as for stand-alone CRAB 7

8 Components!"#$%&"'&"()*+,(-.*/&*0%(1(2!34!"#$%&"'&"()*+,(-.*/&*0%(1(2!34 ( ((((((((((((( (((((( (((((((((5((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#06#7(8#/,*!9:;3<9=<9!9:;3<9=<9 >"*/?+@((%&"'&" >"*/?+@((%&"'&" 8"AB7(/&.&-#+*A0(%&"'*C& 8"AB7(/&.&-#+*A0(%&"'*C& D(E7@"AB7FC&"0FC,( >2G3(H(89IG:( >2G3(H(89IG:( >2G3(H(-.&B&C( >2G3(H(-.&B&C( >.*/&D*0(JG3 >.*/&D*0(JG3 (KAC#.(G73LK(4#+#$#%& KAC#.(G73LK(4#+#$#%& (!AEE#0/(G#0#-&"(!AEE#0/(G#0#-&"( D( D(I0+&"@"&+(CAEE#0/%(#0/(/*%@#+C,(+A(+,&(CAE@A0&0+% I0+&"@"&+(CAEE#0/%(#0/(/*%@#+C,(+A(+,&(CAE@A0&0+% (!"#$%&"'&"(JA"M&"!"#$%&"'&"(JA"M&" D( D(:..(%N$E*%%*A0(#C*+*'*+*&% :..(%N$E*%%*A0(#C*+*'*+*&% (O#%MO"#CM*0- O#%MO"#CM*0- D( D(M&&@(*0?A(#$AN+(+#%M(N0/&"(&B&CN+*A0 M&&@(*0?A(#$AN+(+#%M(N0/&"(&B&CN+*A0 D(*0?A"E(0A+*?*C#+*A0(%&"'*C& D(*0?A"E(0A+*?*C#+*A0(%&"'*C& D(-&0&"#+&("&@A"+(#$AN+(+,&(+#%M D(-&0&"#+&("&@A"+(#$AN+(+,&(+#%M (O#%MK*?&G#0#-&" O#%MK*?&G#0#-&" D( D("&EA'&(+,&(/"A@;AB(#?+&"(+,&(+#%M(*%(?*0*%,&/ "&EA'&(+,&(/"A@;AB(#?+&"(+,&(+#%M(*%(?*0*%,&/ D(E#0#-&(%@#C&(*0(+,&(/"A@;AB D(E#0#-&(%@#C&(*0(+,&(/"A@;AB D(CA0+#C+(0A+*?*C#+*A0(%&"'*C&(#+(+,&(&0/(A?(+#%M D(CA0+#C+(0A+*?*C#+*A0(%&"'*C&(#+(+,&(&0/(A?(+#%M D(E#0#-&(+,&(.*?&+*E&(A?(+,&(+#%M D(E#0#-&(+,&(.*?&+*E&(A?(+,&(+#%M (PA$O"#CM*0- PA$O"#CM*0- D( D(!,&CM(+,&(%+#+N%(A?(+,&(+#%MQPA$!,&CM(+,&(%+#+N%(A?(+,&(+#%MQPA$ D(LN&"7(*0+&"'#.R(PA$%(+A(@A..R(8A..(*0+&"'#.(&+CF D(LN&"7(*0+&"'#.R(PA$%(+A(@A..R(8A..(*0+&"'#.(&+CF (>&+SN+@N+ >&+SN+@N+ D(>&+(AN+@N+(A@&"#+*A0%("&TN&%+&/($7(+,&(PA$O"#CM*0- D(>&+(AN+@N+(A@&"#+*A0%("&TN&%+&/($7(+,&(PA$O"#CM*0-8

9 Client User submits a task to the CRABSERVER Site whitelists and blacklists Input data set Output SE Authentication: user s proxy 9

10 crab.cfg [CMSSW] total_number_of_events=2500 number_of_jobs=25 pset=tutorial.py datasetpath=/relvalprodttbar/sam-mc_42_v12_sam-v1/gen-sim-reco output_file=out_ root [USER] return_data=0 copy_data = 1 storage_element = T2_US_UCSD publish_data = 0 publish_data_name = jletts_data dbs_url_for_publication = [GRID] SE_white_list=T3_US_Colorado [CRAB] scheduler=glidein #scheduler=glite jobtype=cmssw use_server=1 10

11 Job Creation and Submission TaskRegister uses glexec to register the task, and creates a unique proxy for the user (if not already there from the TaskLifetimeManager) CrabServerWorker then creates the jobs, and submits them on behalf of the user to CMS supported Grids. Uses glexec and proxy created previously. 11

12 Overview!"#$%&"'&"()*+,(-.*/&*0%(1(2!34 ( ((((((((((((( (((((( (((((((((5((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#06#7(8#/,* 12

13 Matching DESIRED_SEs = "sedcache.hepgrid.uerj.br,storage01.lcg.cscs.ch,gfe02.grid.hep.ph.ic.ac.uk,maite.iihe.ac.be,dcache-secms.desy.de,cit-se.ultralight.org,cluster142.knu.ac.kr,heplnx204.pp.rl.ac.uk,hepse01.colorado.edu" DESIRED_Gatekeepers = "grid-cr5.desy.de:8443/cream-pbs,grid-cr6.desy.de:8443/creampbs,ceprod04.grid.hep.ph.ic.ac.uk:2119/jobmanager-sge,cit-gatekeeper.ultralight.org:2119/jobmanagercondor,ceprod05.grid.hep.ph.ic.ac.uk:8443/cream-sge,heplnx208.pp.rl.ac.uk:8443/creampbs,heplnx206.pp.rl.ac.uk:8443/cream-pbs,heplnx207.pp.rl.ac.uk:8443/cream-pbs,cream01.iihe.ac.be: 8443/cream-pbs,cluster50.knu.ac.kr:8443/cream-pbs,cream01.lcg.cscs.ch:8443/creampbs,ceprod03.grid.hep.ph.ic.ac.uk:2119/jobmanager-sge,ceprod06.grid.hep.ph.ic.ac.uk:8443/creamsge,grid-cr4.desy.de:8443/cream-pbs,cream02.lcg.cscs.ch:8443/cream-pbs,heposg01.colorado.edu:2119/ jobmanager-sge" MATCH_GLIDEIN_Site = "DESY" MATCH_GLIDEIN_SEs = "dcache-se-cms.desy.de" MATCH_GLIDEIN_Gatekeeper = " pbs cms" MATCH_GLIDEIN_Job_Max_Time = MATCH_GLIDEIN_MaxMemMBs = 2048 MATCH_GLIDEIN_Max_Walltime = MATCH_GLIDEIN_ToRetire =

14 Monitoring TaskTracking - keeps track of registered tasks JobTracking: tracks individual jobs updates CMS Dashboard when jobs complete, calls GetOutput GetOutput - fetches output (e.g. stdout, stderr) of the jobs and makes a summary xml file Note: StageOut is not asynchronous in CRAB2. 14

15 Other Components TaskLifetimeManager - keeps proxy updated Notification - asynchronous notifications to users through MessageService (PA). Called by TaskTracking, jobtracking, TaskLifetimeManager, etc. CommandManager - web service to clients AdminManager - component watchdog 15

16 Other Functions StoreResults service Elevation or user output data sets to global DBS JobMon - detailed job monitoring service running on glidein-mon.t2.ucsd.edu; realtime monitoring (cf. Dashboard, which is mainly historical) 16

17 JobMon ( ((((((((((((( (((((( (((((((((A((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#0B#C(D#/,* 17

18 Future For CRAB2, there is no future... Improvements awaited with CRAB3 Tasks vs. Jobs Further isolation of Grid from the user (and with it all the headaches) Asynchronous Stage Out (largest single source of error) 18

19 References CrabOperations CMSPublic/SWGuideCrab CMS/CrabOperations 19

Tutorial for CMS Users: Data Analysis on the Grid with CRAB

Tutorial for CMS Users: Data Analysis on the Grid with CRAB Tutorial for CMS Users: Data Analysis on the Grid with CRAB Benedikt Mura, Hartmut Stadie Institut für Experimentalphysik, Universität Hamburg September 2nd, 2009 In this part you will learn... 1 how to

More information

CRAB tutorial 08/04/2009

CRAB tutorial 08/04/2009 CRAB tutorial 08/04/2009 Federica Fanzago INFN Padova Stefano Lacaprara INFN Legnaro 1 Outline short CRAB tool presentation hand-on session 2 Prerequisities We expect you know: Howto run CMSSW codes locally

More information

Configuring a glideinwms factory

Configuring a glideinwms factory GlideinWMS Training @ UCSD Configuring a glideinwms factory by Igor Sfiligoi (UCSD) UCSD Nov 8th Factory configuration 1 Refresher Glidein factory The glidein factory knows about the sites and does the

More information

ARC integration for CMS

ARC integration for CMS ARC integration for CMS ARC integration for CMS Erik Edelmann 2, Laurence Field 3, Jaime Frey 4, Michael Grønager 2, Kalle Happonen 1, Daniel Johansson 2, Josva Kleist 2, Jukka Klem 1, Jesper Koivumäki

More information

PoS(ACAT)020. Status and evolution of CRAB. Fabio Farina University and INFN Milano-Bicocca S. Lacaprara INFN Legnaro

PoS(ACAT)020. Status and evolution of CRAB. Fabio Farina University and INFN Milano-Bicocca   S. Lacaprara INFN Legnaro Status and evolution of CRAB University and INFN Milano-Bicocca E-mail: fabio.farina@cern.ch S. Lacaprara INFN Legnaro W. Bacchi University and INFN Bologna M. Cinquilli University and INFN Perugia G.

More information

One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool. D. Mason for CMS Software & Computing

One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool. D. Mason for CMS Software & Computing One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool D. Mason for CMS Software & Computing 1 Going to try to give you a picture of the CMS HTCondor/ glideinwms global pool What s the use case

More information

glideinwms Training Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms Training Glidein internals 1

glideinwms Training Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms Training Glidein internals 1 Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) Glidein internals 1 Refresher glidein_startup the glidein_startup script configures and starts Condor on the worker node Glidein

More information

Shooting for the sky: Testing the limits of condor. HTCondor Week May 2015 Edgar Fajardo On behalf of OSG Software and Technology

Shooting for the sky: Testing the limits of condor. HTCondor Week May 2015 Edgar Fajardo On behalf of OSG Software and Technology Shooting for the sky: Testing the limits of condor 21 May 2015 Edgar Fajardo On behalf of OSG Software and Technology 1 Acknowledgement Although I am the one presenting. This work is a product of a collaborative

More information

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of

More information

BOSCO Architecture. Derek Weitzel University of Nebraska Lincoln

BOSCO Architecture. Derek Weitzel University of Nebraska Lincoln BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple

More information

Batch system usage arm euthen F azo he Z J. B T

Batch system usage arm euthen F azo he Z J. B T Batch system usage 10.11.2010 General stuff Computing wikipage: http://dvinfo.ifh.de Central email address for questions & requests: uco-zn@desy.de Data storage: AFS ( /afs/ifh.de/group/amanda/scratch/

More information

glite Grid Services Overview

glite Grid Services Overview The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) glite Grid Services Overview Antonio Calanducci INFN Catania Joint GISELA/EPIKH School for Grid Site Administrators Valparaiso,

More information

Gridbus Portlets -- USER GUIDE -- GRIDBUS PORTLETS 1 1. GETTING STARTED 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4

Gridbus Portlets -- USER GUIDE --  GRIDBUS PORTLETS 1 1. GETTING STARTED 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4 Gridbus Portlets -- USER GUIDE -- www.gridbus.org/broker GRIDBUS PORTLETS 1 1. GETTING STARTED 2 1.1. PREREQUISITES: 2 1.2. INSTALLATION: 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4 3.1. CREATING

More information

The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS

The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS DESY Computing Seminar Frank Volkmer, M. Sc. Bergische Universität Wuppertal Introduction Hardware Pleiades Cluster

More information

High Throughput WAN Data Transfer with Hadoop-based Storage

High Throughput WAN Data Transfer with Hadoop-based Storage High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San

More information

Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio. (INFN Padova) on behalf of Eurogrid support group

Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio. (INFN Padova) on behalf of Eurogrid support group Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio (INFN Padova) on behalf of Eurogrid support group CDF computing model CDF computing model is based on Central farm

More information

Building Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat)

Building Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) Building Campus HTC Sharing Infrastructures Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) HCC: Campus Grids Motivation We have 3 clusters in 2 cities. Our largest (4400 cores) is

More information

Using ssh as portal The CMS CRAB over glideinwms experience

Using ssh as portal The CMS CRAB over glideinwms experience Journal of Physics: Conference Series OPEN ACCESS Using ssh as portal The CMS CRAB over glideinwms experience To cite this article: S Belforte et al 2014 J. Phys.: Conf. Ser. 513 032006 View the article

More information

Experience with Data-flow, DQM and Analysis of TIF Data

Experience with Data-flow, DQM and Analysis of TIF Data Experience with Data-flow, DQM and Analysis of TIF Data G. Bagliesi, R.J. Bainbridge, T. Boccali, A. Bocci, V. Ciulli, N. De Filippis, M. De Mattia, S. Dutta, D. Giordano, L. Mirabito, C. Noeding, F. Palla,

More information

Condor-G: HTCondor for grid submission. Jaime Frey (UW-Madison), Jeff Dost (UCSD)

Condor-G: HTCondor for grid submission. Jaime Frey (UW-Madison), Jeff Dost (UCSD) Condor-G: HTCondor for grid submission Jaime Frey (UW-Madison), Jeff Dost (UCSD) Acknowledgement These slides are heavily based on the presentation Jaime Frey gave at UCSD in Feb 2011 http://www.t2.ucsd.edu/twiki2/bin/view/main/glideinfactory1111

More information

I Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011

I Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011 I Tier-3 di CMS-Italia: stato e prospettive Claudio Grandi Workshop CCR GRID 2011 Outline INFN Perugia Tier-3 R&D Computing centre: activities, storage and batch system CMS services: bottlenecks and workarounds

More information

The PanDA System in the ATLAS Experiment

The PanDA System in the ATLAS Experiment 1a, Jose Caballero b, Kaushik De a, Tadashi Maeno b, Maxim Potekhin b, Torre Wenaus b on behalf of the ATLAS collaboration a University of Texas at Arlington, Science Hall, PO Box 19059, Arlington, TX

More information

NAF & NUC reports. Y. Kemp for NAF admin team H. Stadie for NUC 4 th annual Alliance Workshop Dresden,

NAF & NUC reports. Y. Kemp for NAF admin team H. Stadie for NUC 4 th annual Alliance Workshop Dresden, NAF & NUC reports Y. Kemp for NAF admin team H. Stadie for NUC 4 th annual Alliance Workshop Dresden, 2.12.2010 NAF: The Overview Picture and current resources Yves Kemp NAF 2.12.2010 Page 2 ATLAS NAF

More information

glideinwms Frontend Installation

glideinwms Frontend Installation glideinwms Training @ UCSD glideinwms Frontend Installation Part 1 Condor Installation by Igor Sfiligoi (UCSD) UCSD Jan 17th 2012 Condor Install 1 Overview Introduction Planning and Common setup Central

More information

Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing

Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing Wolf Behrenhoff, Christoph Wissing DESY Computing Seminar May 17th, 2010 Page 1 Installation of

More information

Introducing the HTCondor-CE

Introducing the HTCondor-CE Introducing the HTCondor-CE CHEP 2015 Presented by Edgar Fajardo 1 Introduction In summer 2012, OSG performed an internal review of major software components, looking for strategic weaknesses. One highlighted

More information

Corral: A Glide-in Based Service for Resource Provisioning

Corral: A Glide-in Based Service for Resource Provisioning : A Glide-in Based Service for Resource Provisioning Gideon Juve USC Information Sciences Institute juve@usc.edu Outline Throughput Applications Grid Computing Multi-level scheduling and Glideins Example:

More information

UW-ATLAS Experiences with Condor

UW-ATLAS Experiences with Condor UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS

More information

Scientific Computing on Emerging Infrastructures. using HTCondor

Scientific Computing on Emerging Infrastructures. using HTCondor Scientific Computing on Emerging Infrastructures using HT HT Week, 20th May 2015 University of California, San Diego 1 Scientific Computing LHC probes nature at 10-17cm Weak Scale Scientific instruments:

More information

glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD)

glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD) Outline A high level overview of the glideinwms Description of the components 2 glideinwms from 10k feet 3 Refresher - HTCondor A Condor pool

More information

BOSCO Architecture. Derek Weitzel University of Nebraska Lincoln

BOSCO Architecture. Derek Weitzel University of Nebraska Lincoln BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple

More information

Configuring Network Zones

Configuring Network Zones Configuring Network Zones Okta offers authentication whitelisting and blacklisting based on zones. Zones are sets of IP address ranges or geolocations or proxy type defined by an admin and are used in

More information

DIRAC pilot framework and the DIRAC Workload Management System

DIRAC pilot framework and the DIRAC Workload Management System Journal of Physics: Conference Series DIRAC pilot framework and the DIRAC Workload Management System To cite this article: Adrian Casajus et al 2010 J. Phys.: Conf. Ser. 219 062049 View the article online

More information

A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations

A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations Yuichi Tsujita Abstract A Windows PC cluster is focused for its high availabilities and fruitful

More information

Introduction. creating job-definition files into structured directories etc.

Introduction. creating job-definition files into structured directories etc. Introduction full atlas simulation chain using Grid tools by Alessandro de Salvo, that provides: environment settings scripts for job definition, submission, jobs handling (cancellation etc.), and getting

More information

A Virtual Comet. HTCondor Week 2017 May Edgar Fajardo On behalf of OSG Software and Technology

A Virtual Comet. HTCondor Week 2017 May Edgar Fajardo On behalf of OSG Software and Technology A Virtual Comet HTCondor Week 2017 May 3 2017 Edgar Fajardo On behalf of OSG Software and Technology 1 Working in Comet What my friends think I do What Instagram thinks I do What my boss thinks I do 2

More information

Distributed production managers meeting. Armando Fella on behalf of Italian distributed computing group

Distributed production managers meeting. Armando Fella on behalf of Italian distributed computing group Distributed production managers meeting Armando Fella on behalf of Italian distributed computing group Distributed Computing human network CNAF Caltech SLAC McGill Queen Mary RAL LAL and Lyon Bari Legnaro

More information

Monitoring tools in EGEE

Monitoring tools in EGEE Monitoring tools in EGEE Piotr Nyczyk CERN IT/GD Joint OSG and EGEE Operations Workshop - 3 Abingdon, 27-29 September 2005 www.eu-egee.org Kaleidoscope of monitoring tools Monitoring for operations Covered

More information

What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018

What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison

More information

An Automated Cluster/Grid Task and Data Management System

An Automated Cluster/Grid Task and Data Management System An Automated Cluster/Grid Task and Data Management System Luís Miranda, Tiago Sá, António Pina, and Vitor Oliveira Department of Informatics, University of Minho, Portugal {lmiranda,tiagosa,pina,vspo}@di.uminho.pt

More information

The CMS workload management system

The CMS workload management system Journal of Physics: Conference Series The CMS workload management system To cite this article: M Cinquilli et al 2012 J. Phys.: Conf. Ser. 396 032113 View the article online for updates and enhancements.

More information

Submitting XML and Excel. Cost Reports

Submitting XML and Excel. Cost Reports Submitting XML and Excel Cost Reports CSDR Submit-Review Submitter Training July 1, 2014 April 5, 2011 Submitter Training The following slides provide step-by-step screenshots to illustrate the actions

More information

How to use computing resources at Grid

How to use computing resources at Grid How to use computing resources at Grid Nikola Grkic ngrkic@ipb.ac.rs Scientific Computing Laboratory Institute of Physics Belgrade, Serbia Academic and Educat ional Gr id Init iat ive of S er bia Oct.

More information

Factory Ops Site Debugging

Factory Ops Site Debugging Factory Ops Site Debugging This talk shows detailed examples of how we debug site problems By Jeff Dost (UCSD) Factory Ops Site Debugging 1 Overview Validation Rundiff Held Waiting Pending Unmatched Factory

More information

ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova

ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova INFN/TC-09/03 May 05, 2009 DESIGN AND IMPLEMENTATION OF THE GLITE CREAM JOB MANAGEMENT SERVICE Cristina Aiftimiei, Paolo Andreetto, Sara Bertocco,

More information

Architecture Proposal

Architecture Proposal Nordic Testbed for Wide Area Computing and Data Handling NORDUGRID-TECH-1 19/02/2002 Architecture Proposal M.Ellert, A.Konstantinov, B.Kónya, O.Smirnova, A.Wäänänen Introduction The document describes

More information

Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY

Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY Journal of Physics: Conference Series OPEN ACCESS Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY To cite this article: Elena Bystritskaya et al 2014 J. Phys.: Conf.

More information

CMS Grid Computing at TAMU Performance, Monitoring and Current Status of the Brazos Cluster

CMS Grid Computing at TAMU Performance, Monitoring and Current Status of the Brazos Cluster CMS Grid Computing at TAMU Performance, Monitoring and Current Status of the Brazos Cluster Vaikunth Thukral Department of Physics and Astronomy Texas A&M University 1 Outline Grid Computing with CMS:

More information

Advanced Job Submission on the Grid

Advanced Job Submission on the Grid Advanced Job Submission on the Grid Antun Balaz Scientific Computing Laboratory Institute of Physics Belgrade http://www.scl.rs/ 30 Nov 11 Dec 2009 www.eu-egee.org Scope User Interface Submit job Workload

More information

g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.

g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac. g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse

More information

High Performance Computing Course Notes Grid Computing I

High Performance Computing Course Notes Grid Computing I High Performance Computing Course Notes 2008-2009 2009 Grid Computing I Resource Demands Even as computer power, data storage, and communication continue to improve exponentially, resource capacities are

More information

The HTCondor CacheD. Derek Weitzel, Brian Bockelman University of Nebraska Lincoln

The HTCondor CacheD. Derek Weitzel, Brian Bockelman University of Nebraska Lincoln The HTCondor CacheD Derek Weitzel, Brian Bockelman University of Nebraska Lincoln Today s Talk Today s talk summarizes work for my a part of my PhD Dissertation Also, this work has been accepted to PDPTA

More information

Using Hadoop File System and MapReduce in a small/medium Grid site

Using Hadoop File System and MapReduce in a small/medium Grid site Journal of Physics: Conference Series Using Hadoop File System and MapReduce in a small/medium Grid site To cite this article: H Riahi et al 2012 J. Phys.: Conf. Ser. 396 042050 View the article online

More information

QosCosGrid Middleware

QosCosGrid Middleware Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space PLGrid Plus QosCosGrid Middleware Domain-oriented services and resources

More information

CMS Belgian T2. G. Bruno UCL, Louvain, Belgium on behalf of the CMS Belgian T2 community. GridKa T1/2 meeting, Karlsruhe Germany February

CMS Belgian T2. G. Bruno UCL, Louvain, Belgium on behalf of the CMS Belgian T2 community. GridKa T1/2 meeting, Karlsruhe Germany February UCL, Louvain, Belgium on behalf of the community GridKa, Karlsruhe Germany February 2 2006 community Institutes in CMS (40 physicists) UA (Antwerp) UCL (Louvain-La-Neuve) ULB (Brussels) UMH (Mons) VUB

More information

Andrea Sciabà CERN, Switzerland

Andrea Sciabà CERN, Switzerland Frascati Physics Series Vol. VVVVVV (xxxx), pp. 000-000 XX Conference Location, Date-start - Date-end, Year THE LHC COMPUTING GRID Andrea Sciabà CERN, Switzerland Abstract The LHC experiments will start

More information

Job submission and management through web services: the experience with the CREAM service

Job submission and management through web services: the experience with the CREAM service Journal of Physics: Conference Series Job submission and management through web services: the experience with the CREAM service To cite this article: C Aiftimiei et al 2008 J. Phys.: Conf. Ser. 119 062004

More information

Why Firewalls? Firewall Characteristics

Why Firewalls? Firewall Characteristics Why Firewalls? Firewalls are effective to: Protect local systems. Protect network-based security threats. Provide secured and controlled access to Internet. Provide restricted and controlled access from

More information

Interconnect EGEE and CNGRID e-infrastructures

Interconnect EGEE and CNGRID e-infrastructures Interconnect EGEE and CNGRID e-infrastructures Giuseppe Andronico Interoperability and Interoperation between Europe, India and Asia Workshop Barcelona - Spain, June 2 2007 FP6 2004 Infrastructures 6-SSA-026634

More information

The EU DataGrid Fabric Management

The EU DataGrid Fabric Management The EU DataGrid Fabric Management The European DataGrid Project Team http://www.eudatagrid.org DataGrid is a project funded by the European Union Grid Tutorial 4/3/2004 n 1 EDG Tutorial Overview Workload

More information

An Automated Cluster/Grid Task and Data Management System

An Automated Cluster/Grid Task and Data Management System An Automated Cluster/Grid Task and Data Management System Luís Miranda, Tiago Sá, António Pina, and Vítor Oliveira Department of Informatics, University of Minho, Portugal {lmiranda,tiagosa,pina,vspo}@di.uminho.pt

More information

THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap

THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap Arnie Miles Georgetown University adm35@georgetown.edu http://thebes.arc.georgetown.edu The Thebes middleware project was

More information

glideinwms UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1

glideinwms UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1 glideinwms Training @ UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1 Regulating User Priorities UCSD Jan 18th 2012 Condor Tunning 2 User priorities By default, the Negotiator

More information

The Global Grid and the Local Analysis

The Global Grid and the Local Analysis The Global Grid and the Local Analysis Yves Kemp DESY IT GridKA School, 11.9.2008 Overview Global and globalization : Some thoughts Anatomy of an analysis and the computing resources needed Boundary between

More information

GROWL Scripts and Web Services

GROWL Scripts and Web Services GROWL Scripts and Web Services Grid Technology Group E-Science Centre r.j.allan@dl.ac.uk GROWL Collaborative project (JISC VRE I programme) between CCLRC Daresbury Laboratory and the Universities of Cambridge

More information

Service Availability Monitor tests for ATLAS

Service Availability Monitor tests for ATLAS Service Availability Monitor tests for ATLAS Current Status Work in progress Alessandro Di Girolamo CERN IT/GS Critical Tests: Current Status Now running ATLAS specific tests together with standard OPS

More information

CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status

CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status High Level Requirements for user analysis computing Code Development Environment Compile, run,

More information

The National Analysis DESY

The National Analysis DESY The National Analysis Facility @ DESY Yves Kemp for the NAF team DESY IT Hamburg & DV Zeuthen 10.9.2008 GridKA School NAF: National Analysis Facility Why a talk about an Analysis Facility at a Grid School?

More information

CMS experience of running glideinwms in High Availability mode

CMS experience of running glideinwms in High Availability mode CMS experience of running glideinwms in High Availability mode I Sfiligoi 1, J Letts 1, S Belforte 2, A McCrea 1, K Larson 3, M Zvada 4, B Holzman 3, P Mhashilkar 3, D C Bradley 5, M D Saiz Santos 1, F

More information

HTCondor Week 2015: Implementing an HTCondor service at CERN

HTCondor Week 2015: Implementing an HTCondor service at CERN HTCondor Week 2015: Implementing an HTCondor service at CERN Iain Steers, Jérôme Belleman, Ulrich Schwickerath IT-PES-PS HTCondor Week 2015 HTCondor at CERN 2 Outline The Move Environment Grid Pilot Local

More information

Monitoring for IT Services and WLCG. Alberto AIMAR CERN-IT for the MONIT Team

Monitoring for IT Services and WLCG. Alberto AIMAR CERN-IT for the MONIT Team Monitoring for IT Services and WLCG Alberto AIMAR CERN-IT for the MONIT Team 2 Outline Scope and Mandate Architecture and Data Flow Technologies and Usage WLCG Monitoring IT DC and Services Monitoring

More information

Large scale commissioning and operational experience with tier-2 to tier-2 data transfer links in CMS

Large scale commissioning and operational experience with tier-2 to tier-2 data transfer links in CMS Journal of Physics: Conference Series Large scale commissioning and operational experience with tier-2 to tier-2 data transfer links in CMS To cite this article: J Letts and N Magini 2011 J. Phys.: Conf.

More information

The glite middleware. Presented by John White EGEE-II JRA1 Dep. Manager On behalf of JRA1 Enabling Grids for E-sciencE

The glite middleware. Presented by John White EGEE-II JRA1 Dep. Manager On behalf of JRA1 Enabling Grids for E-sciencE The glite middleware Presented by John White EGEE-II JRA1 Dep. Manager On behalf of JRA1 John.White@cern.ch www.eu-egee.org EGEE and glite are registered trademarks Outline glite distributions Software

More information

Globus Toolkit 4 Execution Management. Alexandra Jimborean International School of Informatics Hagenberg, 2009

Globus Toolkit 4 Execution Management. Alexandra Jimborean International School of Informatics Hagenberg, 2009 Globus Toolkit 4 Execution Management Alexandra Jimborean International School of Informatics Hagenberg, 2009 2 Agenda of the day Introduction to Globus Toolkit and GRAM Zoom In WS GRAM Usage Guide Architecture

More information

Thebes, WS SAML, and Federation

Thebes, WS SAML, and Federation Thebes, WS SAML, and Federation Internet2 Fall Member Meeting November 3, 2010 Thebes Consortium Georgetown University Arnie Miles adm35@georgetown.edu http://code.google.com/p/thebes/ Back story I haven't

More information

CMS HLT production using Grid tools

CMS HLT production using Grid tools CMS HLT production using Grid tools Flavia Donno (INFN Pisa) Claudio Grandi (INFN Bologna) Ivano Lippi (INFN Padova) Francesco Prelz (INFN Milano) Andrea Sciaba` (INFN Pisa) Massimo Sgaravatto (INFN Padova)

More information

XRAY Grid TO BE OR NOT TO BE?

XRAY Grid TO BE OR NOT TO BE? XRAY Grid TO BE OR NOT TO BE? 1 I was not always a Grid sceptic! I started off as a grid enthusiast e.g. by insisting that Grid be part of the ESRF Upgrade Program outlined in the Purple Book : In this

More information

2/26/2017. For instance, consider running Word Count across 20 splits

2/26/2017. For instance, consider running Word Count across 20 splits Based on the slides of prof. Pietro Michiardi Hadoop Internals https://github.com/michiard/disc-cloud-course/raw/master/hadoop/hadoop.pdf Job: execution of a MapReduce application across a data set Task:

More information

Exam : 1Z Title : Oracle SOA Foundation Practitioner. Version : Demo

Exam : 1Z Title : Oracle SOA Foundation Practitioner. Version : Demo Exam : 1Z1-451 Title : Oracle SOA Foundation Practitioner Version : Demo 1.What is true when implementing human reactions that are part of composite applications using the human task component in SOA 11g?

More information

Securely Access Services Over AWS PrivateLink. January 2019

Securely Access Services Over AWS PrivateLink. January 2019 Securely Access Services Over AWS PrivateLink January 2019 Notices This document is provided for informational purposes only. It represents AWS s current product offerings and practices as of the date

More information

Introduction to SciTokens

Introduction to SciTokens Introduction to SciTokens Brian Bockelman, On Behalf of the SciTokens Team https://scitokens.org This material is based upon work supported by the National Science Foundation under Grant No. 1738962. Any

More information

Troubleshooting Grid authentication from the client side

Troubleshooting Grid authentication from the client side Troubleshooting Grid authentication from the client side By Adriaan van der Zee RP1 presentation 2009-02-04 Contents The Grid @NIKHEF The project Grid components and interactions X.509 certificates, proxies

More information

Interfacing HTCondor-CE with OpenStack: technical questions

Interfacing HTCondor-CE with OpenStack: technical questions Interfacing HTCondor-CE with OpenStack: technical questions Jose Caballero HTCondor Week 2017 Disclaimer facts: This work was done under the umbrella of OSG Technologies Investigations. So there were other

More information

PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM

PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM Szabolcs Pota 1, Gergely Sipos 2, Zoltan Juhasz 1,3 and Peter Kacsuk 2 1 Department of Information Systems, University of Veszprem, Hungary 2 Laboratory

More information

Distributing storage of LHC data - in the nordic countries

Distributing storage of LHC data - in the nordic countries Distributing storage of LHC data - in the nordic countries Gerd Behrmann INTEGRATE ASG Lund, May 11th, 2016 Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed

More information

The SciTokens Authorization Model: JSON Web Tokens & OAuth

The SciTokens Authorization Model: JSON Web Tokens & OAuth The SciTokens Authorization Model: JSON Web Tokens & OAuth Jim Basney Brian Bockelman This material is based upon work supported by the National Science

More information

CMS Computing Model with Focus on German Tier1 Activities

CMS Computing Model with Focus on German Tier1 Activities CMS Computing Model with Focus on German Tier1 Activities Seminar über Datenverarbeitung in der Hochenergiephysik DESY Hamburg, 24.11.2008 Overview The Large Hadron Collider The Compact Muon Solenoid CMS

More information

Comparative evaluation of software tools accessing relational databases from a (real) grid environments

Comparative evaluation of software tools accessing relational databases from a (real) grid environments Comparative evaluation of software tools accessing relational databases from a (real) grid environments Giacinto Donvito, Guido Cuscela, Massimiliano Missiato, Vicenzo Spinoso, Giorgio Maggi INFN-Bari

More information

DESY at the LHC. Klaus Mőnig. On behalf of the ATLAS, CMS and the Grid/Tier2 communities

DESY at the LHC. Klaus Mőnig. On behalf of the ATLAS, CMS and the Grid/Tier2 communities DESY at the LHC Klaus Mőnig On behalf of the ATLAS, CMS and the Grid/Tier2 communities A bit of History In Spring 2005 DESY decided to participate in the LHC experimental program During summer 2005 a group

More information

Scalability and interoperability within glideinwms

Scalability and interoperability within glideinwms Journal of Physics: Conference Series Scalability and interoperability within glideinwms To cite this article: D Bradley et al 2010 J. Phys.: Conf. Ser. 219 062036 View the article online for updates and

More information

Workload Management. Stefano Lacaprara. CMS Physics Week, FNAL, 12/16 April Department of Physics INFN and University of Padova

Workload Management. Stefano Lacaprara. CMS Physics Week, FNAL, 12/16 April Department of Physics INFN and University of Padova Workload Management Stefano Lacaprara Department of Physics INFN and University of Padova CMS Physics Week, FNAL, 12/16 April 2005 Outline 1 Workload Management: the CMS way General Architecture Present

More information

GoDocker. A batch scheduling system with Docker containers

GoDocker. A batch scheduling system with Docker containers GoDocker A batch scheduling system with Docker containers Web - http://www.genouest.org/godocker/ Code - https://bitbucket.org/osallou/go-docker Twitter - #godocker Olivier Sallou IRISA - 2016 CC-BY-SA

More information

Monte Carlo Production on the Grid by the H1 Collaboration

Monte Carlo Production on the Grid by the H1 Collaboration Journal of Physics: Conference Series Monte Carlo Production on the Grid by the H1 Collaboration To cite this article: E Bystritskaya et al 2012 J. Phys.: Conf. Ser. 396 032067 Recent citations - Monitoring

More information

NorduGrid Tutorial. Client Installation and Job Examples

NorduGrid Tutorial. Client Installation and Job Examples NorduGrid Tutorial Client Installation and Job Examples Linux Clusters for Super Computing Conference Linköping, Sweden October 18, 2004 Arto Teräs arto.teras@csc.fi Steps to Start Using NorduGrid 1) Install

More information

An update on the scalability limits of the Condor batch system

An update on the scalability limits of the Condor batch system An update on the scalability limits of the Condor batch system D Bradley 1, T St Clair 1, M Farrellee 1, Z Guo 1, M Livny 1, I Sfiligoi 2, T Tannenbaum 1 1 University of Wisconsin, Madison, WI, USA 2 University

More information

LCG-2 and glite Architecture and components

LCG-2 and glite Architecture and components LCG-2 and glite Architecture and components Author E.Slabospitskaya www.eu-egee.org Outline Enabling Grids for E-sciencE What are LCG-2 and glite? glite Architecture Release 1.0 review What is glite?.

More information

ecamms Overview Pennsylvania Asphalt Pavement Association 55 th Annual Asphalt Paving Conference January 21, 2015 Timothy L. Ramirez, P.E.

ecamms Overview Pennsylvania Asphalt Pavement Association 55 th Annual Asphalt Paving Conference January 21, 2015 Timothy L. Ramirez, P.E. ecamms Overview Pennsylvania Asphalt Pavement Association 55 th Annual Asphalt Paving Conference January 21, 2015 Timothy L. Ramirez, P.E. ecamms Overview Topics What is ecamms? Current Business Partner

More information

Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011)

Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011) Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011) Sergio Maffioletti Grid Computing Competence Centre, University of Zurich http://www.gc3.uzh.ch/

More information

Audience Profile: Course Outline. Introduction & Architecture. SharePoint 2013 Administration Basics. Creating Web Applications and Site Collections

Audience Profile: Course Outline. Introduction & Architecture. SharePoint 2013 Administration Basics. Creating Web Applications and Site Collections Audience Profile: This course is for Consultants and IT Professionals, who need to understand how to plan, implement and configure SharePoint. This course is mainly targeted for administrators and support

More information

Clarens Web Services Architecture

Clarens Web Services Architecture Clarens Web Services Architecture CHEP 2003 March 24-28, Grid Architecture, Infrastructure & Middleware Conrad Steenberg, Eric Aslakson, Julian Bunn, Harvey Newman California Institute of Technology Developed

More information