CRAB 2 and CRABServer at UCSD
|
|
- Vernon Warren
- 6 years ago
- Views:
Transcription
1 CRAB 2 and CRABServer at UCSD JAMES LETTS* (with advice and slides from Sanjay Padhi) - UCSD - * I have never been a CRAB developer or even been responsible for running a CRABServer, so if you see errors in this talk, please jump in and correct me... 1
2 Basics UCSD hosts one of the CRAB Servers for CMS Analysis with CRAB 2 Other servers at CERN, Bari, Desy... From the glidein point-of-view, the CRAB Server is a scheduler (schedd) These slides present some of the features and internal workings of CRAB Server 2
3 Caveat CRAB 2 is effectively legacy software Will be replaced by CRAB 3 (WMAgent) CRAB 2 will be maintained by CMS Analysis Operations* until at least November No significant development effort Emphasis on stability and maintaining the present functionality 3
4 CRAB 2 Modes Stand-alone CRAB, and CRAB Server glite and glidein submitters Users can submit tasks in any combination by appropriate crab.cfg: #scheduler=glite scheduler=glidein use_server=1 4
5 Stand-alone vs. Server The standalone model has the advantage of simplicity and stability... But lacks features provided by a more advanced client-server architecture. CRAB Server is placed between the user and the Grid to perform a set of actions. Automate more the analysis work flow and to improve the scalability of the system. 5
6 Features of CRAB Server Job monitoring with a web-based front end Automatic resubmission of aborted jobs Notification of job progress User support from Analysis Operations hn-cms-crabfeedback@cern.ch General principle to isolate user as much as possible from the Grid: submission, job/task monitoring, output retrieval. 6
7 CRAB 2 Server CRAB Server tries to isolate the user from the Grid User interface is same as for stand-alone CRAB 7
8 Components!"#$%&"'&"()*+,(-.*/&*0%(1(2!34!"#$%&"'&"()*+,(-.*/&*0%(1(2!34 ( ((((((((((((( (((((( (((((((((5((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#06#7(8#/,*!9:;3<9=<9!9:;3<9=<9 >"*/?+@((%&"'&" >"*/?+@((%&"'&" 8"AB7(/&.&-#+*A0(%&"'*C& 8"AB7(/&.&-#+*A0(%&"'*C& D(E7@"AB7FC&"0FC,( >2G3(H(89IG:( >2G3(H(89IG:( >2G3(H(-.&B&C( >2G3(H(-.&B&C( >.*/&D*0(JG3 >.*/&D*0(JG3 (KAC#.(G73LK(4#+#$#%& KAC#.(G73LK(4#+#$#%& (!AEE#0/(G#0#-&"(!AEE#0/(G#0#-&"( D( D(I0+&"@"&+(CAEE#0/%(#0/(/*%@#+C,(+A(+,&(CAE@A0&0+% I0+&"@"&+(CAEE#0/%(#0/(/*%@#+C,(+A(+,&(CAE@A0&0+% (!"#$%&"'&"(JA"M&"!"#$%&"'&"(JA"M&" D( D(:..(%N$E*%%*A0(#C*+*'*+*&% :..(%N$E*%%*A0(#C*+*'*+*&% (O#%MO"#CM*0- O#%MO"#CM*0- D( D(M&&@(*0?A(#$AN+(+#%M(N0/&"(&B&CN+*A0 M&&@(*0?A(#$AN+(+#%M(N0/&"(&B&CN+*A0 D(*0?A"E(0A+*?*C#+*A0(%&"'*C& D(*0?A"E(0A+*?*C#+*A0(%&"'*C& D(-&0&"#+&("&@A"+(#$AN+(+,&(+#%M D(-&0&"#+&("&@A"+(#$AN+(+,&(+#%M (O#%MK*?&G#0#-&" O#%MK*?&G#0#-&" D( D("&EA'&(+,&(/"A@;AB(#?+&"(+,&(+#%M(*%(?*0*%,&/ "&EA'&(+,&(/"A@;AB(#?+&"(+,&(+#%M(*%(?*0*%,&/ D(E#0#-&(%@#C&(*0(+,&(/"A@;AB D(E#0#-&(%@#C&(*0(+,&(/"A@;AB D(CA0+#C+(0A+*?*C#+*A0(%&"'*C&(#+(+,&(&0/(A?(+#%M D(CA0+#C+(0A+*?*C#+*A0(%&"'*C&(#+(+,&(&0/(A?(+#%M D(E#0#-&(+,&(.*?&+*E&(A?(+,&(+#%M D(E#0#-&(+,&(.*?&+*E&(A?(+,&(+#%M (PA$O"#CM*0- PA$O"#CM*0- D( D(!,&CM(+,&(%+#+N%(A?(+,&(+#%MQPA$!,&CM(+,&(%+#+N%(A?(+,&(+#%MQPA$ D(LN&"7(*0+&"'#.R(PA$%(+A(@A..R(8A..(*0+&"'#.(&+CF D(LN&"7(*0+&"'#.R(PA$%(+A(@A..R(8A..(*0+&"'#.(&+CF (>&+SN+@N+ >&+SN+@N+ D(>&+(AN+@N+(A@&"#+*A0%("&TN&%+&/($7(+,&(PA$O"#CM*0- D(>&+(AN+@N+(A@&"#+*A0%("&TN&%+&/($7(+,&(PA$O"#CM*0-8
9 Client User submits a task to the CRABSERVER Site whitelists and blacklists Input data set Output SE Authentication: user s proxy 9
10 crab.cfg [CMSSW] total_number_of_events=2500 number_of_jobs=25 pset=tutorial.py datasetpath=/relvalprodttbar/sam-mc_42_v12_sam-v1/gen-sim-reco output_file=out_ root [USER] return_data=0 copy_data = 1 storage_element = T2_US_UCSD publish_data = 0 publish_data_name = jletts_data dbs_url_for_publication = [GRID] SE_white_list=T3_US_Colorado [CRAB] scheduler=glidein #scheduler=glite jobtype=cmssw use_server=1 10
11 Job Creation and Submission TaskRegister uses glexec to register the task, and creates a unique proxy for the user (if not already there from the TaskLifetimeManager) CrabServerWorker then creates the jobs, and submits them on behalf of the user to CMS supported Grids. Uses glexec and proxy created previously. 11
12 Overview!"#$%&"'&"()*+,(-.*/&*0%(1(2!34 ( ((((((((((((( (((((( (((((((((5((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#06#7(8#/,* 12
13 Matching DESIRED_SEs = "sedcache.hepgrid.uerj.br,storage01.lcg.cscs.ch,gfe02.grid.hep.ph.ic.ac.uk,maite.iihe.ac.be,dcache-secms.desy.de,cit-se.ultralight.org,cluster142.knu.ac.kr,heplnx204.pp.rl.ac.uk,hepse01.colorado.edu" DESIRED_Gatekeepers = "grid-cr5.desy.de:8443/cream-pbs,grid-cr6.desy.de:8443/creampbs,ceprod04.grid.hep.ph.ic.ac.uk:2119/jobmanager-sge,cit-gatekeeper.ultralight.org:2119/jobmanagercondor,ceprod05.grid.hep.ph.ic.ac.uk:8443/cream-sge,heplnx208.pp.rl.ac.uk:8443/creampbs,heplnx206.pp.rl.ac.uk:8443/cream-pbs,heplnx207.pp.rl.ac.uk:8443/cream-pbs,cream01.iihe.ac.be: 8443/cream-pbs,cluster50.knu.ac.kr:8443/cream-pbs,cream01.lcg.cscs.ch:8443/creampbs,ceprod03.grid.hep.ph.ic.ac.uk:2119/jobmanager-sge,ceprod06.grid.hep.ph.ic.ac.uk:8443/creamsge,grid-cr4.desy.de:8443/cream-pbs,cream02.lcg.cscs.ch:8443/cream-pbs,heposg01.colorado.edu:2119/ jobmanager-sge" MATCH_GLIDEIN_Site = "DESY" MATCH_GLIDEIN_SEs = "dcache-se-cms.desy.de" MATCH_GLIDEIN_Gatekeeper = " pbs cms" MATCH_GLIDEIN_Job_Max_Time = MATCH_GLIDEIN_MaxMemMBs = 2048 MATCH_GLIDEIN_Max_Walltime = MATCH_GLIDEIN_ToRetire =
14 Monitoring TaskTracking - keeps track of registered tasks JobTracking: tracks individual jobs updates CMS Dashboard when jobs complete, calls GetOutput GetOutput - fetches output (e.g. stdout, stderr) of the jobs and makes a summary xml file Note: StageOut is not asynchronous in CRAB2. 14
15 Other Components TaskLifetimeManager - keeps proxy updated Notification - asynchronous notifications to users through MessageService (PA). Called by TaskTracking, jobtracking, TaskLifetimeManager, etc. CommandManager - web service to clients AdminManager - component watchdog 15
16 Other Functions StoreResults service Elevation or user output data sets to global DBS JobMon - detailed job monitoring service running on glidein-mon.t2.ucsd.edu; realtime monitoring (cf. Dashboard, which is mainly historical) 16
17 JobMon ( ((((((((((((( (((((( (((((((((A((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((3#0B#C(D#/,* 17
18 Future For CRAB2, there is no future... Improvements awaited with CRAB3 Tasks vs. Jobs Further isolation of Grid from the user (and with it all the headaches) Asynchronous Stage Out (largest single source of error) 18
19 References CrabOperations CMSPublic/SWGuideCrab CMS/CrabOperations 19
Tutorial for CMS Users: Data Analysis on the Grid with CRAB
Tutorial for CMS Users: Data Analysis on the Grid with CRAB Benedikt Mura, Hartmut Stadie Institut für Experimentalphysik, Universität Hamburg September 2nd, 2009 In this part you will learn... 1 how to
More informationCRAB tutorial 08/04/2009
CRAB tutorial 08/04/2009 Federica Fanzago INFN Padova Stefano Lacaprara INFN Legnaro 1 Outline short CRAB tool presentation hand-on session 2 Prerequisities We expect you know: Howto run CMSSW codes locally
More informationConfiguring a glideinwms factory
GlideinWMS Training @ UCSD Configuring a glideinwms factory by Igor Sfiligoi (UCSD) UCSD Nov 8th Factory configuration 1 Refresher Glidein factory The glidein factory knows about the sites and does the
More informationARC integration for CMS
ARC integration for CMS ARC integration for CMS Erik Edelmann 2, Laurence Field 3, Jaime Frey 4, Michael Grønager 2, Kalle Happonen 1, Daniel Johansson 2, Josva Kleist 2, Jukka Klem 1, Jesper Koivumäki
More informationPoS(ACAT)020. Status and evolution of CRAB. Fabio Farina University and INFN Milano-Bicocca S. Lacaprara INFN Legnaro
Status and evolution of CRAB University and INFN Milano-Bicocca E-mail: fabio.farina@cern.ch S. Lacaprara INFN Legnaro W. Bacchi University and INFN Bologna M. Cinquilli University and INFN Perugia G.
More informationOne Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool. D. Mason for CMS Software & Computing
One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool D. Mason for CMS Software & Computing 1 Going to try to give you a picture of the CMS HTCondor/ glideinwms global pool What s the use case
More informationglideinwms Training Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms Training Glidein internals 1
Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) Glidein internals 1 Refresher glidein_startup the glidein_startup script configures and starts Condor on the worker node Glidein
More informationShooting for the sky: Testing the limits of condor. HTCondor Week May 2015 Edgar Fajardo On behalf of OSG Software and Technology
Shooting for the sky: Testing the limits of condor 21 May 2015 Edgar Fajardo On behalf of OSG Software and Technology 1 Acknowledgement Although I am the one presenting. This work is a product of a collaborative
More information30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy
Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of
More informationBOSCO Architecture. Derek Weitzel University of Nebraska Lincoln
BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple
More informationBatch system usage arm euthen F azo he Z J. B T
Batch system usage 10.11.2010 General stuff Computing wikipage: http://dvinfo.ifh.de Central email address for questions & requests: uco-zn@desy.de Data storage: AFS ( /afs/ifh.de/group/amanda/scratch/
More informationglite Grid Services Overview
The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) glite Grid Services Overview Antonio Calanducci INFN Catania Joint GISELA/EPIKH School for Grid Site Administrators Valparaiso,
More informationGridbus Portlets -- USER GUIDE -- GRIDBUS PORTLETS 1 1. GETTING STARTED 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4
Gridbus Portlets -- USER GUIDE -- www.gridbus.org/broker GRIDBUS PORTLETS 1 1. GETTING STARTED 2 1.1. PREREQUISITES: 2 1.2. INSTALLATION: 2 2. AUTHENTICATION 3 3. WORKING WITH PROJECTS 4 3.1. CREATING
More informationThe Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS
The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS DESY Computing Seminar Frank Volkmer, M. Sc. Bergische Universität Wuppertal Introduction Hardware Pleiades Cluster
More informationHigh Throughput WAN Data Transfer with Hadoop-based Storage
High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San
More informationEurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio. (INFN Padova) on behalf of Eurogrid support group
Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio (INFN Padova) on behalf of Eurogrid support group CDF computing model CDF computing model is based on Central farm
More informationBuilding Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat)
Building Campus HTC Sharing Infrastructures Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) HCC: Campus Grids Motivation We have 3 clusters in 2 cities. Our largest (4400 cores) is
More informationUsing ssh as portal The CMS CRAB over glideinwms experience
Journal of Physics: Conference Series OPEN ACCESS Using ssh as portal The CMS CRAB over glideinwms experience To cite this article: S Belforte et al 2014 J. Phys.: Conf. Ser. 513 032006 View the article
More informationExperience with Data-flow, DQM and Analysis of TIF Data
Experience with Data-flow, DQM and Analysis of TIF Data G. Bagliesi, R.J. Bainbridge, T. Boccali, A. Bocci, V. Ciulli, N. De Filippis, M. De Mattia, S. Dutta, D. Giordano, L. Mirabito, C. Noeding, F. Palla,
More informationCondor-G: HTCondor for grid submission. Jaime Frey (UW-Madison), Jeff Dost (UCSD)
Condor-G: HTCondor for grid submission Jaime Frey (UW-Madison), Jeff Dost (UCSD) Acknowledgement These slides are heavily based on the presentation Jaime Frey gave at UCSD in Feb 2011 http://www.t2.ucsd.edu/twiki2/bin/view/main/glideinfactory1111
More informationI Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011
I Tier-3 di CMS-Italia: stato e prospettive Claudio Grandi Workshop CCR GRID 2011 Outline INFN Perugia Tier-3 R&D Computing centre: activities, storage and batch system CMS services: bottlenecks and workarounds
More informationThe PanDA System in the ATLAS Experiment
1a, Jose Caballero b, Kaushik De a, Tadashi Maeno b, Maxim Potekhin b, Torre Wenaus b on behalf of the ATLAS collaboration a University of Texas at Arlington, Science Hall, PO Box 19059, Arlington, TX
More informationNAF & NUC reports. Y. Kemp for NAF admin team H. Stadie for NUC 4 th annual Alliance Workshop Dresden,
NAF & NUC reports Y. Kemp for NAF admin team H. Stadie for NUC 4 th annual Alliance Workshop Dresden, 2.12.2010 NAF: The Overview Picture and current resources Yves Kemp NAF 2.12.2010 Page 2 ATLAS NAF
More informationglideinwms Frontend Installation
glideinwms Training @ UCSD glideinwms Frontend Installation Part 1 Condor Installation by Igor Sfiligoi (UCSD) UCSD Jan 17th 2012 Condor Install 1 Overview Introduction Planning and Common setup Central
More informationInstallation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing
Installation of CMSSW in the Grid DESY Computing Seminar May 17th, 2010 Wolf Behrenhoff, Christoph Wissing Wolf Behrenhoff, Christoph Wissing DESY Computing Seminar May 17th, 2010 Page 1 Installation of
More informationIntroducing the HTCondor-CE
Introducing the HTCondor-CE CHEP 2015 Presented by Edgar Fajardo 1 Introduction In summer 2012, OSG performed an internal review of major software components, looking for strategic weaknesses. One highlighted
More informationCorral: A Glide-in Based Service for Resource Provisioning
: A Glide-in Based Service for Resource Provisioning Gideon Juve USC Information Sciences Institute juve@usc.edu Outline Throughput Applications Grid Computing Multi-level scheduling and Glideins Example:
More informationUW-ATLAS Experiences with Condor
UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS
More informationScientific Computing on Emerging Infrastructures. using HTCondor
Scientific Computing on Emerging Infrastructures using HT HT Week, 20th May 2015 University of California, San Diego 1 Scientific Computing LHC probes nature at 10-17cm Weak Scale Scientific instruments:
More informationglideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD)
glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD) Outline A high level overview of the glideinwms Description of the components 2 glideinwms from 10k feet 3 Refresher - HTCondor A Condor pool
More informationBOSCO Architecture. Derek Weitzel University of Nebraska Lincoln
BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple
More informationConfiguring Network Zones
Configuring Network Zones Okta offers authentication whitelisting and blacklisting based on zones. Zones are sets of IP address ranges or geolocations or proxy type defined by an admin and are used in
More informationDIRAC pilot framework and the DIRAC Workload Management System
Journal of Physics: Conference Series DIRAC pilot framework and the DIRAC Workload Management System To cite this article: Adrian Casajus et al 2010 J. Phys.: Conf. Ser. 219 062049 View the article online
More informationA Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations
A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations Yuichi Tsujita Abstract A Windows PC cluster is focused for its high availabilities and fruitful
More informationIntroduction. creating job-definition files into structured directories etc.
Introduction full atlas simulation chain using Grid tools by Alessandro de Salvo, that provides: environment settings scripts for job definition, submission, jobs handling (cancellation etc.), and getting
More informationA Virtual Comet. HTCondor Week 2017 May Edgar Fajardo On behalf of OSG Software and Technology
A Virtual Comet HTCondor Week 2017 May 3 2017 Edgar Fajardo On behalf of OSG Software and Technology 1 Working in Comet What my friends think I do What Instagram thinks I do What my boss thinks I do 2
More informationDistributed production managers meeting. Armando Fella on behalf of Italian distributed computing group
Distributed production managers meeting Armando Fella on behalf of Italian distributed computing group Distributed Computing human network CNAF Caltech SLAC McGill Queen Mary RAL LAL and Lyon Bari Legnaro
More informationMonitoring tools in EGEE
Monitoring tools in EGEE Piotr Nyczyk CERN IT/GD Joint OSG and EGEE Operations Workshop - 3 Abingdon, 27-29 September 2005 www.eu-egee.org Kaleidoscope of monitoring tools Monitoring for operations Covered
More informationWhat s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018
What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison
More informationAn Automated Cluster/Grid Task and Data Management System
An Automated Cluster/Grid Task and Data Management System Luís Miranda, Tiago Sá, António Pina, and Vitor Oliveira Department of Informatics, University of Minho, Portugal {lmiranda,tiagosa,pina,vspo}@di.uminho.pt
More informationThe CMS workload management system
Journal of Physics: Conference Series The CMS workload management system To cite this article: M Cinquilli et al 2012 J. Phys.: Conf. Ser. 396 032113 View the article online for updates and enhancements.
More informationSubmitting XML and Excel. Cost Reports
Submitting XML and Excel Cost Reports CSDR Submit-Review Submitter Training July 1, 2014 April 5, 2011 Submitter Training The following slides provide step-by-step screenshots to illustrate the actions
More informationHow to use computing resources at Grid
How to use computing resources at Grid Nikola Grkic ngrkic@ipb.ac.rs Scientific Computing Laboratory Institute of Physics Belgrade, Serbia Academic and Educat ional Gr id Init iat ive of S er bia Oct.
More informationFactory Ops Site Debugging
Factory Ops Site Debugging This talk shows detailed examples of how we debug site problems By Jeff Dost (UCSD) Factory Ops Site Debugging 1 Overview Validation Rundiff Held Waiting Pending Unmatched Factory
More informationISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova
ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova INFN/TC-09/03 May 05, 2009 DESIGN AND IMPLEMENTATION OF THE GLITE CREAM JOB MANAGEMENT SERVICE Cristina Aiftimiei, Paolo Andreetto, Sara Bertocco,
More informationArchitecture Proposal
Nordic Testbed for Wide Area Computing and Data Handling NORDUGRID-TECH-1 19/02/2002 Architecture Proposal M.Ellert, A.Konstantinov, B.Kónya, O.Smirnova, A.Wäänänen Introduction The document describes
More informationMonitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY
Journal of Physics: Conference Series OPEN ACCESS Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY To cite this article: Elena Bystritskaya et al 2014 J. Phys.: Conf.
More informationCMS Grid Computing at TAMU Performance, Monitoring and Current Status of the Brazos Cluster
CMS Grid Computing at TAMU Performance, Monitoring and Current Status of the Brazos Cluster Vaikunth Thukral Department of Physics and Astronomy Texas A&M University 1 Outline Grid Computing with CMS:
More informationAdvanced Job Submission on the Grid
Advanced Job Submission on the Grid Antun Balaz Scientific Computing Laboratory Institute of Physics Belgrade http://www.scl.rs/ 30 Nov 11 Dec 2009 www.eu-egee.org Scope User Interface Submit job Workload
More informationg-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.
g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse
More informationHigh Performance Computing Course Notes Grid Computing I
High Performance Computing Course Notes 2008-2009 2009 Grid Computing I Resource Demands Even as computer power, data storage, and communication continue to improve exponentially, resource capacities are
More informationThe HTCondor CacheD. Derek Weitzel, Brian Bockelman University of Nebraska Lincoln
The HTCondor CacheD Derek Weitzel, Brian Bockelman University of Nebraska Lincoln Today s Talk Today s talk summarizes work for my a part of my PhD Dissertation Also, this work has been accepted to PDPTA
More informationUsing Hadoop File System and MapReduce in a small/medium Grid site
Journal of Physics: Conference Series Using Hadoop File System and MapReduce in a small/medium Grid site To cite this article: H Riahi et al 2012 J. Phys.: Conf. Ser. 396 042050 View the article online
More informationQosCosGrid Middleware
Domain-oriented services and resources of Polish Infrastructure for Supporting Computational Science in the European Research Space PLGrid Plus QosCosGrid Middleware Domain-oriented services and resources
More informationCMS Belgian T2. G. Bruno UCL, Louvain, Belgium on behalf of the CMS Belgian T2 community. GridKa T1/2 meeting, Karlsruhe Germany February
UCL, Louvain, Belgium on behalf of the community GridKa, Karlsruhe Germany February 2 2006 community Institutes in CMS (40 physicists) UA (Antwerp) UCL (Louvain-La-Neuve) ULB (Brussels) UMH (Mons) VUB
More informationAndrea Sciabà CERN, Switzerland
Frascati Physics Series Vol. VVVVVV (xxxx), pp. 000-000 XX Conference Location, Date-start - Date-end, Year THE LHC COMPUTING GRID Andrea Sciabà CERN, Switzerland Abstract The LHC experiments will start
More informationJob submission and management through web services: the experience with the CREAM service
Journal of Physics: Conference Series Job submission and management through web services: the experience with the CREAM service To cite this article: C Aiftimiei et al 2008 J. Phys.: Conf. Ser. 119 062004
More informationWhy Firewalls? Firewall Characteristics
Why Firewalls? Firewalls are effective to: Protect local systems. Protect network-based security threats. Provide secured and controlled access to Internet. Provide restricted and controlled access from
More informationInterconnect EGEE and CNGRID e-infrastructures
Interconnect EGEE and CNGRID e-infrastructures Giuseppe Andronico Interoperability and Interoperation between Europe, India and Asia Workshop Barcelona - Spain, June 2 2007 FP6 2004 Infrastructures 6-SSA-026634
More informationThe EU DataGrid Fabric Management
The EU DataGrid Fabric Management The European DataGrid Project Team http://www.eudatagrid.org DataGrid is a project funded by the European Union Grid Tutorial 4/3/2004 n 1 EDG Tutorial Overview Workload
More informationAn Automated Cluster/Grid Task and Data Management System
An Automated Cluster/Grid Task and Data Management System Luís Miranda, Tiago Sá, António Pina, and Vítor Oliveira Department of Informatics, University of Minho, Portugal {lmiranda,tiagosa,pina,vspo}@di.uminho.pt
More informationTHEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap
THEBES: THE GRID MIDDLEWARE PROJECT Project Overview, Status Report and Roadmap Arnie Miles Georgetown University adm35@georgetown.edu http://thebes.arc.georgetown.edu The Thebes middleware project was
More informationglideinwms UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1
glideinwms Training @ UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1 Regulating User Priorities UCSD Jan 18th 2012 Condor Tunning 2 User priorities By default, the Negotiator
More informationThe Global Grid and the Local Analysis
The Global Grid and the Local Analysis Yves Kemp DESY IT GridKA School, 11.9.2008 Overview Global and globalization : Some thoughts Anatomy of an analysis and the computing resources needed Boundary between
More informationGROWL Scripts and Web Services
GROWL Scripts and Web Services Grid Technology Group E-Science Centre r.j.allan@dl.ac.uk GROWL Collaborative project (JISC VRE I programme) between CCLRC Daresbury Laboratory and the Universities of Cambridge
More informationService Availability Monitor tests for ATLAS
Service Availability Monitor tests for ATLAS Current Status Work in progress Alessandro Di Girolamo CERN IT/GS Critical Tests: Current Status Now running ATLAS specific tests together with standard OPS
More informationCMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status
CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status High Level Requirements for user analysis computing Code Development Environment Compile, run,
More informationThe National Analysis DESY
The National Analysis Facility @ DESY Yves Kemp for the NAF team DESY IT Hamburg & DV Zeuthen 10.9.2008 GridKA School NAF: National Analysis Facility Why a talk about an Analysis Facility at a Grid School?
More informationCMS experience of running glideinwms in High Availability mode
CMS experience of running glideinwms in High Availability mode I Sfiligoi 1, J Letts 1, S Belforte 2, A McCrea 1, K Larson 3, M Zvada 4, B Holzman 3, P Mhashilkar 3, D C Bradley 5, M D Saiz Santos 1, F
More informationHTCondor Week 2015: Implementing an HTCondor service at CERN
HTCondor Week 2015: Implementing an HTCondor service at CERN Iain Steers, Jérôme Belleman, Ulrich Schwickerath IT-PES-PS HTCondor Week 2015 HTCondor at CERN 2 Outline The Move Environment Grid Pilot Local
More informationMonitoring for IT Services and WLCG. Alberto AIMAR CERN-IT for the MONIT Team
Monitoring for IT Services and WLCG Alberto AIMAR CERN-IT for the MONIT Team 2 Outline Scope and Mandate Architecture and Data Flow Technologies and Usage WLCG Monitoring IT DC and Services Monitoring
More informationLarge scale commissioning and operational experience with tier-2 to tier-2 data transfer links in CMS
Journal of Physics: Conference Series Large scale commissioning and operational experience with tier-2 to tier-2 data transfer links in CMS To cite this article: J Letts and N Magini 2011 J. Phys.: Conf.
More informationThe glite middleware. Presented by John White EGEE-II JRA1 Dep. Manager On behalf of JRA1 Enabling Grids for E-sciencE
The glite middleware Presented by John White EGEE-II JRA1 Dep. Manager On behalf of JRA1 John.White@cern.ch www.eu-egee.org EGEE and glite are registered trademarks Outline glite distributions Software
More informationGlobus Toolkit 4 Execution Management. Alexandra Jimborean International School of Informatics Hagenberg, 2009
Globus Toolkit 4 Execution Management Alexandra Jimborean International School of Informatics Hagenberg, 2009 2 Agenda of the day Introduction to Globus Toolkit and GRAM Zoom In WS GRAM Usage Guide Architecture
More informationThebes, WS SAML, and Federation
Thebes, WS SAML, and Federation Internet2 Fall Member Meeting November 3, 2010 Thebes Consortium Georgetown University Arnie Miles adm35@georgetown.edu http://code.google.com/p/thebes/ Back story I haven't
More informationCMS HLT production using Grid tools
CMS HLT production using Grid tools Flavia Donno (INFN Pisa) Claudio Grandi (INFN Bologna) Ivano Lippi (INFN Padova) Francesco Prelz (INFN Milano) Andrea Sciaba` (INFN Pisa) Massimo Sgaravatto (INFN Padova)
More informationXRAY Grid TO BE OR NOT TO BE?
XRAY Grid TO BE OR NOT TO BE? 1 I was not always a Grid sceptic! I started off as a grid enthusiast e.g. by insisting that Grid be part of the ESRF Upgrade Program outlined in the Purple Book : In this
More information2/26/2017. For instance, consider running Word Count across 20 splits
Based on the slides of prof. Pietro Michiardi Hadoop Internals https://github.com/michiard/disc-cloud-course/raw/master/hadoop/hadoop.pdf Job: execution of a MapReduce application across a data set Task:
More informationExam : 1Z Title : Oracle SOA Foundation Practitioner. Version : Demo
Exam : 1Z1-451 Title : Oracle SOA Foundation Practitioner Version : Demo 1.What is true when implementing human reactions that are part of composite applications using the human task component in SOA 11g?
More informationSecurely Access Services Over AWS PrivateLink. January 2019
Securely Access Services Over AWS PrivateLink January 2019 Notices This document is provided for informational purposes only. It represents AWS s current product offerings and practices as of the date
More informationIntroduction to SciTokens
Introduction to SciTokens Brian Bockelman, On Behalf of the SciTokens Team https://scitokens.org This material is based upon work supported by the National Science Foundation under Grant No. 1738962. Any
More informationTroubleshooting Grid authentication from the client side
Troubleshooting Grid authentication from the client side By Adriaan van der Zee RP1 presentation 2009-02-04 Contents The Grid @NIKHEF The project Grid components and interactions X.509 certificates, proxies
More informationInterfacing HTCondor-CE with OpenStack: technical questions
Interfacing HTCondor-CE with OpenStack: technical questions Jose Caballero HTCondor Week 2017 Disclaimer facts: This work was done under the umbrella of OSG Technologies Investigations. So there were other
More informationPARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM
PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM Szabolcs Pota 1, Gergely Sipos 2, Zoltan Juhasz 1,3 and Peter Kacsuk 2 1 Department of Information Systems, University of Veszprem, Hungary 2 Laboratory
More informationDistributing storage of LHC data - in the nordic countries
Distributing storage of LHC data - in the nordic countries Gerd Behrmann INTEGRATE ASG Lund, May 11th, 2016 Agenda WLCG: A world wide computing grid for the LHC NDGF: The Nordic Tier 1 dcache: Distributed
More informationThe SciTokens Authorization Model: JSON Web Tokens & OAuth
The SciTokens Authorization Model: JSON Web Tokens & OAuth Jim Basney Brian Bockelman This material is based upon work supported by the National Science
More informationCMS Computing Model with Focus on German Tier1 Activities
CMS Computing Model with Focus on German Tier1 Activities Seminar über Datenverarbeitung in der Hochenergiephysik DESY Hamburg, 24.11.2008 Overview The Large Hadron Collider The Compact Muon Solenoid CMS
More informationComparative evaluation of software tools accessing relational databases from a (real) grid environments
Comparative evaluation of software tools accessing relational databases from a (real) grid environments Giacinto Donvito, Guido Cuscela, Massimiliano Missiato, Vicenzo Spinoso, Giorgio Maggi INFN-Bari
More informationDESY at the LHC. Klaus Mőnig. On behalf of the ATLAS, CMS and the Grid/Tier2 communities
DESY at the LHC Klaus Mőnig On behalf of the ATLAS, CMS and the Grid/Tier2 communities A bit of History In Spring 2005 DESY decided to participate in the LHC experimental program During summer 2005 a group
More informationScalability and interoperability within glideinwms
Journal of Physics: Conference Series Scalability and interoperability within glideinwms To cite this article: D Bradley et al 2010 J. Phys.: Conf. Ser. 219 062036 View the article online for updates and
More informationWorkload Management. Stefano Lacaprara. CMS Physics Week, FNAL, 12/16 April Department of Physics INFN and University of Padova
Workload Management Stefano Lacaprara Department of Physics INFN and University of Padova CMS Physics Week, FNAL, 12/16 April 2005 Outline 1 Workload Management: the CMS way General Architecture Present
More informationGoDocker. A batch scheduling system with Docker containers
GoDocker A batch scheduling system with Docker containers Web - http://www.genouest.org/godocker/ Code - https://bitbucket.org/osallou/go-docker Twitter - #godocker Olivier Sallou IRISA - 2016 CC-BY-SA
More informationMonte Carlo Production on the Grid by the H1 Collaboration
Journal of Physics: Conference Series Monte Carlo Production on the Grid by the H1 Collaboration To cite this article: E Bystritskaya et al 2012 J. Phys.: Conf. Ser. 396 032067 Recent citations - Monitoring
More informationNorduGrid Tutorial. Client Installation and Job Examples
NorduGrid Tutorial Client Installation and Job Examples Linux Clusters for Super Computing Conference Linköping, Sweden October 18, 2004 Arto Teräs arto.teras@csc.fi Steps to Start Using NorduGrid 1) Install
More informationAn update on the scalability limits of the Condor batch system
An update on the scalability limits of the Condor batch system D Bradley 1, T St Clair 1, M Farrellee 1, Z Guo 1, M Livny 1, I Sfiligoi 2, T Tannenbaum 1 1 University of Wisconsin, Madison, WI, USA 2 University
More informationLCG-2 and glite Architecture and components
LCG-2 and glite Architecture and components Author E.Slabospitskaya www.eu-egee.org Outline Enabling Grids for E-sciencE What are LCG-2 and glite? glite Architecture Release 1.0 review What is glite?.
More informationecamms Overview Pennsylvania Asphalt Pavement Association 55 th Annual Asphalt Paving Conference January 21, 2015 Timothy L. Ramirez, P.E.
ecamms Overview Pennsylvania Asphalt Pavement Association 55 th Annual Asphalt Paving Conference January 21, 2015 Timothy L. Ramirez, P.E. ecamms Overview Topics What is ecamms? Current Business Partner
More informationGrid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011)
Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011) Sergio Maffioletti Grid Computing Competence Centre, University of Zurich http://www.gc3.uzh.ch/
More informationAudience Profile: Course Outline. Introduction & Architecture. SharePoint 2013 Administration Basics. Creating Web Applications and Site Collections
Audience Profile: This course is for Consultants and IT Professionals, who need to understand how to plan, implement and configure SharePoint. This course is mainly targeted for administrators and support
More informationClarens Web Services Architecture
Clarens Web Services Architecture CHEP 2003 March 24-28, Grid Architecture, Infrastructure & Middleware Conrad Steenberg, Eric Aslakson, Julian Bunn, Harvey Newman California Institute of Technology Developed
More information