Experimental Computing. Frank Porter System Manager: Juan Barayoga

Transcription

1 Experimental Computing Frank Porter System Manager: Juan Barayoga 1 Frank Porter, Caltech DoE Review, July 21, 2004

2 2 Frank Porter, Caltech DoE Review, July 21, 2004

3 HEP Experimental Computing System Description (I) CPU farm Linux on dual-intel-cpu rack-mounted units, 1-2 GByte memory each Currently 122 CPUs in farm PBS batch system for resource allocation 100BaseT network connection to each unit KVM switches for local keyboard/mouse/video File servers Also Linux/Intel-based IDE RAID 5 technology 7.9 TByte capacity on five servers Gbit ethernet NFS, AFS, Samba file serving software 3 Frank Porter, Caltech DoE Review, July 21, 2004

4 HEP Experimental Computing System Description (II) Interactive servers Desktop mixture of linux and Windows on Intel Central interactive linux servers (4 dual-cpu), legacy AIX servers Recent purchase of eight high performance desktops (including local RAID 0 serial ATA) for heavy interactive analysis Other CPU services NT domain servers (primary and secondary) Web servers (Linux and NT) Objectivity (for BaBar) Tape drives DLT tape library on fileserver DLT drive on NT Mostly used for backups now 4 Frank Porter, Caltech DoE Review, July 21, 2004

5 HEP Experimental Computing System Description (III) Network 100BaseT capability available everywhere, 2 subnets for security, capacity HEP gigabit ethernet switches, plus CITNET 2000 Wireless 11 Mbps, maintained by Caltech WAN supported by Caltech Printers (principally 2 color and 2 B&W) VGA projectors (conference rooms, plus roamer) UPS for critical services (network, file servers, mail server, web server) 5 Frank Porter, Caltech DoE Review, July 21, 2004

6 6 Frank Porter, Caltech DoE Review, July 21, 2004

7 Caltech is a major site for BaBar Monte Carlo Production Two jobs on each of 40 dual-piii nodes; Four jobs (hyperthreading mode) on each of 20 dual Xeon nodes. Each job has 512 MB RAM available. Allocations Signal modes (large variety) Generics: B 0 B 0, B + B, c c, uds, τ + τ, µ + µ Alex Samuel runs MC production at Caltech Checks 1 2 times/day Request new allocation every 2 3 weeks Consecutive allocations run overlapped, so no dead time except when drain queue to upgrade conditions, background triggers, or software Currently, third most productive BaBar site for SP6. 7 Frank Porter, Caltech DoE Review, July 21, 2004

8 Caltech BaBar Monte Carlo Production 8 Frank Porter, Caltech DoE Review, July 21, 2004

9 Caltech BaBar Monte Carlo Production, Weekly Stats Stats from BaBar Monte Carlo production sites in past week (July 16, 2004): Total events produced in SP6: M Site Runs Done Runs Failed Failure Rate (%) Events (M) Machines Events/Machine Cpu Eff. (%) Site eff. (%) uvic utd caltech osu cuboulder albany tud ccin2p utenn infn fzk uk-spgrid uvic slac westgrid uofl Frank Porter, Caltech DoE Review, July 21, 2004

10 BaBar Physics Analysis at Caltech Use of Caltech computing for BaBar data analysis is increasing Off-loads SLAC computing Using both batch queues and interactive analysis Set up for full CM2-based user analysis Have new releases Can import datasets from SLAC Can compile/debug/run user code; building and running is generally faster than running at SLAC Used for large (100 s of GB) physics dataset storage (eg, ntuples) and analysis 10 Frank Porter, Caltech DoE Review, July 21, 2004

11 MINOS Looming Large Pre-data MINOS uses farm for occasional Monte Carlo runs, some analysis. Beam expected to start in December. Computing model remains somewhat unclear relative to Caltech, but likely will mean something like: Some reprocessing of data Monte Carlo production, possibly substantial Physics analysis, requiring efficient interactive access to data A worry: FNAL is not going with Red Hat Enterprise Linux. 11 Frank Porter, Caltech DoE Review, July 21, 2004

12 Continuous Evolution Farm continues to grow to match needs. In 2004 installed additional 20 dual-cpu machines with 2.8 GHz Intel Xeon processors. Currently configuring order for another 20. Haven t hit any scaling limit yet. Blade servers: opted against so far, will continue to watch. Rack space, power/heat load issues. Recently added electrical capacity, will have to do more. Cost/unit approximately constant, performance/unit increases. Disk space continues to grow to match needs. Cost/unit approximately constant, performance/unit increases. Replaced DQS batch system with PBS. No remaining reliance on AIX for services. 12 Frank Porter, Caltech DoE Review, July 21, 2004

13 Continuous Evolution (II) Disk space and network replacing tapes; tapes required mostly for backups. Recently upgraded UPS; still need a bit more. Additional switches (Cisco 2948G, 2970) recently purchased, currently have four 13 Frank Porter, Caltech DoE Review, July 21, 2004

14 Caltech Support Caltech s ITS (Information Technology Services) provides variety of services benefitting HEP Site-wide software license agreements These agreements have improved considerably over time, and are now quite flexible in permitting desired uses. Autocad, Visio, Pro Engineer Maple, Mathematica, Matlab Microsoft: OS, Office, Visual Studio, Project Norton antivirus (NAV) PCTeX SSH, WinSCP Adobe Acrobat New: Red Hat Enterprise Linux 14 Frank Porter, Caltech DoE Review, July 21, 2004

15 Caltech Support Networking Networking support Campus backbone, and equipment/maintenance for WAN connection provided by Caltech Caltech provides ISP (cable modem, PPP, ISDN) service Caltech monitors security alerts Major Caltech campus-wide network upgrade Uniform wireless b (partial) implementation 15 Frank Porter, Caltech DoE Review, July 21, 2004

16 Operations New farm and CPU-servers were brought up with help from BaBar, and largely restricted to BaBar as guinea pigs. Now supporting all of the groups. However, use continues to be dominated by BaBar with heavy Monte Carlo and analysis demands. Decision to go uniformly with Linux was largely the result of personnel concerns. [Phase-out of AIX is essentially complete.] Juan Barayoga is system manager. Additional help from part-time students, physicists. 16 Frank Porter, Caltech DoE Review, July 21, 2004

17 Comments on Caltech HEP computing budget Computing requirements are increasing rapidly, at same time power/dollar increases. Operating budget request is made up of two pieces; Maintenance and system administration, almost entirely salary. Equipment budget. Actually supersedes most former maintenance expense, with continuous upgrade replacement more effective than repair. Proposal is to maintain current level of funding. 17 Frank Porter, Caltech DoE Review, July 21, 2004