WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect

Transcription

1 WISE: Big Data, Little Money - Lessons Learned Tim Conrow WISE System Architect GRITS III June 17, 2011

2 The Problem Heavy ops processing load R/W >3TB/day, Sustained 1Gbit/s network load 180 TB ops archive (non-irsa) Compressed budget and schedule <$400k for ops system at launch Major hardware purchase at L-6 mo.s Simultaneous reprocessing Context for talk: Low cost, high performance. If you ve got big bucks, a lot of this is irrelevant. Total HW $2M for full mission, inc. s/w licences, backup, IRSA, support contracts, etc. Philosophy: Trade-off of low budget against risk of degraded data quality. Pipe+backup+IRSA +WMOPS all X2 2

3 The Solution? Sun servers fronting JBOD s, S/W RAID-Z Local network mediated by a Cisco switch stack (trunked 1Gbit/s to servers) 32 node compute cluster of 8-core Xeon servers Use cluster-local storage to off-load network Backup via separate i/f 3

4 (The Diagram) I think this should be 3750 X G for the 10Gbit/s i/f Thanks Wendy and Marcy! 4

5 (The Details) 5 Sun X4150 Servers (4 ops + spare) 10 Sun J4400 JBOD s, SAS+RAID-Z, 24TB (18TB usable) each (2 per server) 32 node compute cluster; Dell Xeon PE1950, R610 (2 x quad-core, 12GB, 500GB) 2 Cisco 48-port Catalyst 3750 switches RHE4, Solaris, ZFS, Condor, Ganglia, NFS3 5

6 The Result Supported required throughput No data loss (Yay!) Met processing goals BUT, considerable time and effort spent resolving H/W issues, which we would rather have spent elsewhere WISE and ISG engineers, WISE and IPAC management 6

7 The Blowback A selection of problems, not all. There s also NFS tuning, LDAP, etc. (A lot went right too.) Collision/Retran cascades => network stalls Traced to use of trunked network ports Server drops/freezes under load Multiple rounds of patches and work-arounds from Sun (then Oracle) Frequent spindle failures (really transient SCSI i/f errors) JBOD HBA update + other changes Probably server-related, not just the switch. Most H/W problems were OS/ firmware issues. We did replace a The JBODs on there own caused little trouble. 7

8 The Problem 2 Now, do it all again, 3X faster! It s that compressed schedule thing again. No time for H/W issues. 8

9 The Solution 2 Flirted with NetApp Upgrade switches to replace trunked ports with 10Gbit/s i/f (this happened pre-ops) Upgrade servers, add disk, h/w RAID More compute nodes, more memory Smaller partitions Better load balancing. Local accounts for ops. Subnet re-org. 9

10 The Result 2 Running at 3.3 ops days/day 7 mo. cryo mission done in 2 mo.s 15TB/day network I/O W/R 8TB, 15M files/day No problems no downtime no stress Yay! Thanks ops team! Thanks ISG! 10

11 The Lessons Lessons are specifically for when walking in the risky neighborhood of the price vs. performance space. Mind the server performance envelope Performance and configurations not seen in the wild could be risky Some supported configurations weren t. E.g. multiplexing. Have a large test dataset and schedule for prolonged ops-like run scenarios with ops hardware Have a backup server! We caught some problems pre ops. Not just extra disk. Saved us a lot of trouble. NGC2174, Monkey Nebula 11

12 The Lessons Keep it local and under control Segregate from core router Had multiple performance issue when were were routing through the core A busy network can cause LDAP timeouts Local LDAP server or local ops accounts Restrict non-ops access Big laptop transfers tripped certain bugs that could interfere with dev Have good network and cluster monitoring and debugging tools Seagull Nebula Team engineers being able to monitor/debug saves time 12

13 The Lessons The cheap commodity cluster was awesome Lots of fungible parts = no worries Used local disk to offload network Condor works great. Almost too smart. Try a cluster filesystem next time? May not match WISE-like data access and throughput requirements at a budget price. 13

14 The Lessons Massive backups are massively painful Do you really want conventional tape backups? Really? During reprocessing, backups couldn t keep up - Regeneration and/or replication Fill smaller partitions in sequence then freeze and dump Walking file tree can be where a lot of backup time is spent Lambda Orionis 14

15 The Lessons Who you gonna call? Thanks Rosanne, Wendy et al! ISG, purchasing, vendor contacts on speed dial Management may need to elevate issue priority with vendor Local (or not so local) experts Gregory Dubois-Felsmann and Steffan Luitz (SLAC) reviewed our revised network plan Thanks Roc, George, etc. Comet 65P/Gunn 15

16 Make time for tuning... NFS LDAP File size and count The Lessons Common namespace Data layout and load balancing Mounts vs. symlinks Subnet layout; ops, dev, backup, desktop isolation 16

17 (The Future?) Could a WISE-like mission work with computing resources shared among projects? The cloud? Do missions with high/low data rates, operational intensity mix well? What would have happened if our problems occurred on a multimission platform? QOS contracts at what cost? IC342, The Hidden Galaxy 17