WSDC Hardware Architecture

Transcription

1 WSDC Hardware Architecture Tim Conrow, Lead Engineer Heidi Brandenburg IPAC/Caltech HB 1

2 Overview Hardware System Architecture as presented at the Critical Design Review RFA from CDR board Additional tasks since CDR Current (and currently planned) Hardware System Architecture Status & Deployment Timeline Backup Resource Reporting Testing with v2 Open Issues HB 2

3 WSDC Data Flow White Sands JPL UCLA Images HRP Ingest Server H/K Ancillary data MOS Caltech/WSDC Maneuvers Web SOC Mission Status Science Team Expectant Public Ingest Ops Archive QA IRSA Archive Scan/Frame Pipelines Multi-Frame Pipeline - Level-0,1b frames - Coadds - Frame index - Calibration data - Ancillary data - 2MASS ref. catalog - QA Data - Meta-data Archive SSOID FPG -Working Source DB. Coadd. Frame (epoch) - QA Meta-data - Image Meta-data - Final Products. Catalogs. Atlas HB 3

4 CDR Hardware Plan - Overview HB 4

5 CDR - Cluster ~25 Dual Quad-core Commodity Dell servers. 1U, 8 GB RAM, 500GB SATA storage Some machines have better resources: more RAM/faster CPUS RHEL4 Current frame pipeline scales with increasing cores and increasing number of cluster nodes * * Scales as long as we can push data in and out fast enough, of course. HB 5

6 CDR - Operations Archive The operations archive hosts the WSDC software tree and functions as persistent data store for L0, L1, and L3 products Accessed by pipelines to retrieve their inputs and push their (minimal) products Critically important that the operations archive can support the data rate required to meet the mission operations scenario. Currently we calculate this as 2Gbs. HB 6

7 CDR - Backup Current backup provided by IPAC Systems Group Future backup of operations archive occurs on WISE hardware Utilize hot swappable disk for local, cycling backup Utilize LTO for long term offsite backup (Telemetry, expanded L0 products, Archive) HB 7

8 RFAs At the CDR we received two RFAs relevant to this review: RFA #5 Hardware architecture Concern: Hardware Architecture is at PDR level Recommended Action: Suggest a delta peer level review of hardware architecture in mid 2009 to review final configuration and projected system loading RFA #8 Hardware system development Concern: The plan to construct the WSDC hardware system is not adequately concrete. The team needs a plan that tracks the system resource use. That plan can then be used to construct the hardware system based on recorded usage patterns. Recommended Action: (summary) The system engineer should specify a resource allocation budget to each system module, including the Exec infrastructure. Resource allocations should include CPU use and memory at the very least. Other system constraints, such as I/O required to access RAID storage, should be considered. HB 8

9 Additional Tasks since CDR 6 Month Mission Extension (The Extended Mission) Double the mission length to achieve a second pass over the whole sky Doubles the operations archive size and doubles the backup size Near Earth Object Identification & Tracking (NEO-WISE) Hardware and budgets in planning Preliminary estimate anticipates a 1 in 7 increase in the number of nodes, which currently means 5 machines HB 9

10 Current Hardware Plan - Overview IPAC Perimeter Network Backup Services IPAC Backup Infrastructure IPAC Internal Network Science Team SOC, MOS, Science Team Public SFTP HTTPS HTTP Test Data Archive web application server (QA, Science Issue Tracking) Webserver NFS WISE Switch Backup Services Operations Archive NFS WISE Subnet White Sands, MOS FastCopy SFTP Ingest Perimeter Network Firewall Internal Network Firewall Cluster HB 10

11 Current Hardware Plan - Data Flow & Throughput 25 GB/day Raw Telemetry in 4 Downlinks Ingest Images Housekeeping White Sands MOS L0: 13 MB/FS -> 100 GB/day L1: 75 MB/FS -> 570 GB/day L3: <100 GB/day Operations Archive 150 TB Total (7 month mission) 800 GB/day uncompressed Cluster IPAC Backup Infrastructure 150 TB Total (7 month mission) HB 11

12 Current Plan - Cluster Same as CDR plan with more and better ~35 Dual Quad-core Commodity Dell servers. 1U, 16 GB RAM, 500GB SATA storage RHEL4 Storage is temporary work space Max: 130 GB/day/node Min: 75 GB/day/node Kernel HB 12

13 Current Plan - Operations Archive Same as CDR plan with more and better ~160 TB RAIDed storage controlled by 4 File Servers Cluster Cisco 3750 Network Switch Sun X4150 Sun X4150 Sun X4150 Sun X4150 J4400 J4500 J4400 J4500 J4400 J4500 J4400 J4500 HB 13

14 Current Plan - Backup Future backup of operations archive occurs on WISE hardware Backup during entire mission will take place on the shared IPAC backup infrastructure, run by the IPAC Systems Group More Information on Backup in later slides HB 14

15 Current Plan - Perimeter Network Services Public Internet HTTP HTTPS FastCopy IPAC Perimeter Network Public Web Wiki QAReports (web application) Science Issue Tracking (web application) Ingest Backup Services Mailserver NFS (read only) IPAC Internal Network Not covered at CDR Test Data Archive is an Xserve attached Xserve RAID storage Other services hosted on commodity Linux servers, interchangeable with cluster nodes Disk backup via IPAC backup infrastructure FTP Test Data Archive HB 15

16 Status & Timeline - Cluster & Archive 20 nodes to be installed upstairs week of March 23, 2009 Extant 12 nodes (11 workers + 1 master) moved from downstairs at later date 140 TB + 3 Sun X4150 Fileservers to be installed upstairs by beginning of April Extant 20 TB of storage & fileservers moved upstairs at later date. All WSDC hardware installed or moved by April 1. These deployments complete minimum pre-launch build out of cluster and operations archive HB 16

17 Status & Timeline - Perimeter Network Services Ingest server, Test data archive, public web pages, and internal wiki currently deployed Science Issue Tracking and QA hosted on same machine, nominally of same configuration as a cluster node, installed 5/1/09 to support application deployment. Science Issue Tracking web application deployed 5/15/09 (WSDC Project milestone) QA web application deployed as part of v3.0 integration, test, & delivery. HB 17

18 Backup Raw telemetry archive 25 GB/day spread over 4 deliveries (average) Copy kept on disk at White Sands throughout mission Kept on disk at IPAC throughout mission 3 tape archive copies; at IPAC, elsewhere at CIT, off-campus Manual backups, once/day Hardware: TBD (Advise?) Options DAT: matched to daily volume; no need to rotate media LTO: much used at IPAC already; need to rotate media Redundant units, separate from ops archive backup units Roll out by July 09 HB 18

19 Backup Critical software & reference data is mirrored offsite Ops archive backup ~800 GB/day uncompressed Kept on-line throughout mission Existing IPAC disk-to-disk backup with LTO-3 (400 GB/tape) tape library archive Takes about 6 hours to backup or restore one day s products ~4 tapes written per day: one pair stored elsewhere at CIT No time for full tape backups; all incrementals System commissioned by July 09 No node disk backup HB 19

20 Backup Disaster recovery Off-site mini-cluster able to perform minimal processing for 30 days Ingest and raw telemetry backup L0 archive creation Quicklook (<5% of data processed) and minimal QA Should require no more than Two 8-core Dell 1950 s 2 tape drives (?) <10 TB disk (1 RAID-5 array) Recovery process Run from mini-cluster up to 1 month Rebuild primary cluster Restore old ops DB as processing is taken up by new cluster Full, detailed plan TBD HB 20

21 Resource Reporting All pipelines create log files with resource usage tags: >>>> END iam=>'spawn_instruframecal', host=>'wcnode12', pid=>" ", starttime=>'09/03/10_22:17:34z', >>>>+ endtime=>'09/03/10_22:17:51z', status=>0, signal=>0, retcode=>0, upcpu=>0.43, spcpu=>0.08, uccpu=>7.56, sccpu=>3.14, >>>>+ totcpu=>11.21, util=>0.713, elapt=>16.260, ttime=>0, load=>0, rssk=>146500, rssmb=> , rdmb=>0.000, wtmb=>0.000, >>>>+ rxmb=>34.166, txmb=>3.085, pkcoll=>0 Tag info comes from standard system calls and kernel data structures (accessed via Perl module Sys::Statistics::Linux) Log files are parseable with an internal API. RscRpt, a Perl program, is used after pipeline runs to aggregate resource information from the log and print a report. RscRpt can aggregate information for an entire scan worth of log files, or even multiple scans. TODO: include resource reporting in the integration & test plan HB 21

22 Testing ScanFrame in v2 Hardware for test Cluster at 1/3 size (11 nodes + master) 1 Fileserver attached to 1 disk array Data 8 scans from a full-orbit data simulation Each scan trimmed to 1/3 nominal size - ~84 frames Method Launch all 8 scans concurrently After run, process log files with RscRpt HB 22

23 Testing ScanFrame in v2 Results at 1/3 scale Ingest (1 delivery): ~0.6 hours 8-scan (delivery) run time: ~1 hour Max. memory usage (1 process): ~1GB Max. network rate: ~1Gbps (saturated ~10 min.s) Typical network rate: ~0.1Gbps (sustained) 1 scan run time: ~1 hour (8 at a time) 1 frame run time: ~6 min (88 at a time) Ingest+scans for day complete in < 8 hours HB 23

24 Resource Usage - ScanFrame \starttime = '09/03/06_21:03:22Z' \endtime = '09/03/06_22:03:34Z' \rssk = executable num_entries mean-elapt mean-util mean-totcpu max-rssk max-rdmb max-wtmb max-rxmb max-txmb c i r r r r r r r r null null null null null null null null null null spawn_awaic spawn_fpcal spawn_instrufram spawn_mdet spawn_wsspipepos spawn_wphot spawn_frameqa spawn_flag spawn_sfprex spawn_wssflag spawn_spcal spawn_get_latent HB 24

25 Open Issues Unknown processing requirements for the extended mission and NEO-WISE Pipeline performance on real sky instead of simulations Administering clusters is not easy. We d like to have better tools. We d like better tools for monitoring network performance. HB 25

26 Lessons Learned The commercial FORTRAN compiler from Intel produces significantly faster executables for the xeon processors than the free gfortran. Multiple obscure, hard to debug, problems when using Linux as an NFS server. We re in-process of migrating the last of our busy filesystems to Solaris servers. The various disk arrays have different sweet spots for file sizes. It s necessary to tune arrays for the average file size of our products. ISG has been a big help in getting our storage operating as efficiently as possible. HB 26