High Performance Computing Data Management. Philippe Trautmann BDM High Performance Computing Global Research

Transcription

1 High Performance Computing Management Philippe Trautmann BDM High Performance Computing Global Research

2 HPC Market and Trends High Performance Computing: Availability/Sharing is key European Digital Preservation of Research Output initiative Conclusions 2 Sun Confidential: Partner Under NDA Only

3 HPC Market Overview IDC HPC Application/Industry Forecast Servers Application Segment University Academic Govt. Lab Bio Sciences CAE Defense EDA DCC & Distribution Geosciences & Geo Engineering Weather Economics /Financial Chemical Engineering Other Mechanical Design & Drafting Total Revenue 2009 ($K) $1,800,235 $1,425,431 $1,217,297 $952,761 $871,585 $613,729 $576,228 $529,772 $371,260 $261,750 $223,468 $182,756 $106,400 $9,132,672 IDC Server Revenue by Vendor 2008 Storage 2013 ($K) $2,337,419 $1,863,896 $1,781,031 $1,562,311 $1,186,212 $948,920 $835,046 $807,039 $545,329 $421,115 $260,900 $140,644 $98,205 $12,788,067 CAGR (07-13) 6.75% 6.93% 9.98% 13.16% 8.01% 11.51% 9.72% 11.10% 10.09% 12.62% 3.95% -6.34% -1.98% 4.10% 2009 ($K) % of Mkt $571, % $433, % $652, % $455, % $414, % $173, % $269, % $222, % $119, % $64, % $88, % $20, % $27, % $3,512, % HP IBM Dell Other Sun SUN Other HP DELL IBM IDC Estimates that for every $ spent on Servers An additional $.39 is spent on storage An additional $.25 is spent on services Server Revenue by IDC Competitive Segments Segment Supercomputer Division Department Workgroup Price Range $500K and up $250K - $500K $100k - $250K <100K 2009 TAM $B $2.58 $1.30 $3.62 $1.73 CAGR (07 13) 3.20% 1.60% 7.10% 1.90% Sun Confidential: Partner Under NDA Only DOWN SIDE CAGR 1.50% -0.70% -0.04% -0.06% 3

4 The Importance of HPC Organizations are Under Pressure Reduce costs and increase efficiency Improve quality and be first to market Make better and faster decisions Applications becoming increasingly computationally intensive Required to run more and more of these applications Need to analyze more and more data HPC can solve these problems and is now a required technology to stay competitive Sun Confidential: Partner Under NDA Only 4

5 Barriers to High Performance Computing The P in HPC Technical limitations system, storage, interconnect, complexity Exploding Requirements Increasing fidelity of modeling and simulation Instruments that spit out PetaBytes of Requirement for collaborative research Complexity of Use Need reliable solutions that are easy to architect, deploy and use Space, power and cooling issues 5

6 Barriers to HPC Access 2009 Time to Store Time to Compute Exponential Growth 2011 Time to Compute Time to Store Time to Load Time to Load You can only compute as fast as you can move the data 6

7 Barriers to HPC: I/O Bottlenecks Application Enemy #1 Prevents applications from scaling Leads to poor overall application performance Complex CPU? Memory? Storage? Interconnect? Application? Removing I/O Bottlenecks requires an end-toend approach 7

8 A European survey (May 2009) Where do researchers store their data External web service Other Don't store data Digital archive of disciplin Digital archive of organisation Journal Computer at home Organisational server Portable storage carrier Computer at work 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 Source: PARSE Insight Interim report, May 2009 PARSE: Permanent Access to the Records of Science in Europe, EU funded project 8

9 The Information Infrastructure the researcher acts through ingest and access Archival Creation The Body of Knowledge Virtual Research Environment Access Curation Services the researcher shouldn t have to Network worry about the information infrastructure Storage Compute Information Infrastructure 9

10 Current view Distinct Infrastructures / Distinct User Experiences Raw Analysis Analysed Publication Publications Analysis Analysed Publication Publications Analysis Analysed Publication Publications Facility 1 Raw Facility 2 Raw Facility 3 10

11 Future view Common Infrastructure / Common User Experience Raw Catalogue Raw Analysis Analysed Catalogue Publication Catalogue Publication s Catalogue Analysis Analysed Publication Publications Analysis Analysed Publication Publications Analysis Analysed Publication Publications Facility 1 Raw Facility 2 Raw Facility 3 Capacity Storage Standards/ Converters Repositories Publications Repositories 11

12 PARSE Permanent Access to the Records of Science in Europe European funded project 2 years from Closely linked with European Alliance for Permanent Access Roadmap of Science Infrastructure Based on UK s Digital Curation Centre There is a need for a common European Storage Standard UK s UK Research Storage Service pilot funding just agreed 12

13 management and sharing: Big issues stated by the EU Energy requirements Who protects the data ad eterna as publications are linked Terrorism Nation speaking unto nation or project interlinking with project Lack of true large scale project management experience Protectionism 13

14 Flash Technologies accelerate applications Flash Facts > CPU & memory ~ 260 times faster than disk drives > One SSD provides IOPs equal to 100 disk drives at less than 1/500th of the power Flash accelerates I/O, reduces job times and enables more work with less hardware... 14

15 Sun end-to-end infrastructure optimized to accelerate data centric HPC workfl ows M9 QDR Infi niband Network Storage Archive Storage Sun Storage 7000 Unifi ed Storage System High Availability, Manageability, Shared Parallel Storage Access Sun Lustre Storage System Home Directories, Application Code High Performance Parallel File System Input, Results Files Ongoing Computation Sun StorageTek Tape Archives Economic, Green, Long Term Retention Protection of IP Assets SAM Storage Archive Manager HSM 15

16 Petascale projects in the real world TACC 579 TFLOPS World s Largest General Purpose Compute Cluster Sun Constellation X Petabytes 72 GB/sec total bandwidth X systems 800 cores Sun Blade ,936 blades 15,744 CPUs 62,976 cores 125 TB/RAM Switch 3,456 Dual redundant 110 Tb/sec bisectional bandwidth 16

17 INNOVATION MATTERS!! Peta FLOP Computing key points Compute density > Flops/watt, TB/watt, GB/sec I/O technologies > In CPUs, on mother boards, on systems (Flash, SSDs, etc.) > management technologies > I/O technologies Power and Cooling > Density and efficiency Management > > > > > Hardware (provisioning, upgrade & monitoring) Software (OS, application and patching) Job (scheduling and monitoring) services (Scratch, archival, multi-site, etc.) People and procedures Serviceability Sun Confidential CDA Required 17

18 Sun Microsystems, Inc. Sun Confidential CDA Required Sun Confidential CDA Required High Performance Computing Management A European Perspective