High Performance and Cloud Computing (HPCC) for Bioinformatics

Transcription

1 High Performance and Cloud Computing (HPCC) for Bioinformatics King Jordan Georgia Tech January 13, 2016 Adopted From BIOS-ICGEB HPCC for Bioinformatics 1

2 Outline High performance computing (HPC) Cloud computing HPC vs. Cloud computing Cloud computing for bioinformatics January 13,

3 HPC Overview: Client-server architecture January 13,

4 HPC Overview: Supercomputer clusters A computer cluster is a single logical unit consisting of multiple computers that are linked through a local area network (LAN). The networked computers essentially act as a single, much more powerful machine. A computer cluster provides much faster processing speed, larger storage capacity, better data integrity, superior reliability and wider availability of resources. Computer clusters are, however, much more costly to implement and maintain. This results in much higher running overhead compared to a single computer. (This is where cloud computing comes in ) January 13,

5 HPC Overview: Parallel computing Parallel computing is a type of computing architecture in which several processors execute or process an application or computation simultaneously. Parallel computing helps in performing large computations by dividing the workload between more than one processor, all of which work through the computation at the same time. Most supercomputers employ parallel computing principles to operate. Parallel computing is also known as parallel processing. January 13,

6 GA Tech: PACE (Partnership for an Advanced Computing Environment) 1,200 nodes with 30,000 CPU cores 90 terabytes of memory 2 Petabytes of online commodity storage 215 terabytes of high-performance scratch storage January 13,

7 What is Cloud Computing? How is it related to HPC? How does it differ from traditional HPC? January 13,

8 What is Cloud Computing (skeptical view) "The interesting thing about cloud computing is that we've redefined cloud computing to include everything that we already do. I can't think of anything that isn't cloud computing with all of these announcements. The computer industry is the only industry that is more fashion-driven than women's fashion. Maybe I'm an idiot, but I have no idea what anyone is talking about. What is it? It's complete gibberish. It's insane. When is this idiocy going to stop?" Larry Ellison, CEO Oracle, OracleWorld Paul Hodor B A H January 13,

9 Moving towards a more specific definition of Cloud Computing In 2011 the National Institute of Standards and Technology (NIST) issued Special Publication , "The NIST definition of cloud computing Intended as a means for broad comparisons of cloud services and deployment strategies to provide a baseline for discussion on what cloud computing is and how it is used Defines the following categories of concepts Essential characteristics Service models Deployment models Paul Hodor B A H January 13,

10 Essential characteristics of cloud computing (NIST) On-demand self-service Broad network access Resource pooling Rapid elasticity Measured service Paul Hodor B A H January 13,

11 Service models of Cloud Computing (NIST) Software as a Service (SaaS) The capability to use the provider's applications remotely over the network. The user does not manage the server, operating system, storage, even application capabilities. Platform as a Service (PaaS) The capability to deploy and use user-created or acquired applications on infrastructure made available by the provider. The user has control over deployed applications and their configuration, but does not manage servers, operating system, or storage. Infrastructure as a Service (IaaS) The capability to provision computing resources, storage networking, on which to deploy arbitrary software. The user has virtual control over all resources, but does not have control over the physical infrastructure. Paul Hodor B A H January 13,

12 Service models of Cloud Computing (NIST) Private cloud Community cloud Public cloud Hybrid cloud Paul Hodor B A H January 13,

13 Cloud Computing can also be considered as a kind of Commodity Computing Use of large numbers of already-available computing components for parallel computing, to get the greatest amount of useful computation at low cost. Computing done in commodity computers as opposed to high-cost supercomputers or boutique computers Commodity computers are computer systems manufactured by multiple vendors, incorporating components based on open standards Such systems are said to be based on commodity components, since the standardization process promotes lower costs and less differentiation among vendors' products January 13,

14 Cloud Computing was made possible by the convergence of three existing technologies The internet Research on packet networking funded in the 1960s TCP/IP introduced in the 1980s Opening to commercial traffic Virtualization Early work by IBM in the 1960s Hardware virtualization becomes mainstream in the early 2000s Parallel computing First multi- processor computers in the 1960s Birth of the Message Passing Interface (MPI) in 1992 MapReduce paper published in 2004 Paul Hodor B A H January 13,

15 HPC versus Cloud Computing Models Traditional HPC model (Physical data center) Buy a bunch of server boxes Add hard drives for storage Connect servers with cables into an intranet Install an operating system and applications Log in remotely and start working ssh user@mydomain.com Paul Hodor B A H January 13,

16 HPC versus Cloud Computing Models Traditional HPC model (Physical data center) Buy a bunch of server boxes Add hard drives for storage Connect servers with cables into an intranet Install an operating system and applications Log in remotely and start working ssh user@mydomain.com Cloud Computing model (Virtual data center) Provision a bunch of instances Attach virtual volumes for storage Create a virtual private cloud Launch a machine image Log in remotely and start working ssh user@mydomain.com Paul Hodor B A H January 13,

17 Cloud computing: Available platforms Lavanya Rishishwar GATech January 13,

18 Cloud computing: Available platforms Amazon Web Services - Microsoft Azure - Google App Engine - Illumina BaseSpace - IBM Cloud Computing - HP Eucalyptus - HP Cloud - Rackspace Cloud - DigitalOcean CenturyLink Cloud - Verizon Cloud - Computer Sciences Corporation - Virtustream - VMware - Fujitsu Cloud - Dimension Data Cloud - GoGrid - Joyent - Lavanya Rishishwar GATech January 13,

19 Ability to execute Cloud computing: Performance comparison Completeness of vision Gartner Magic Quadrant of Cloud IaaS, 2014 January 13,

20 Cloud computing for bioinformatics Basics & need for cloud computing Barriers to use Widely used platforms Amazon Web Services Microsoft Azure Bionimbus Galaxy Google Illumina BaseSpace January 13,

21 January 13,

22 January 13,

23 January 13,

24 MapReduce Framework with Hadoop [More from Ahsan Huda] January 13,

25 January 13,

26 January 13,

27 January 13,

28 January 13,

29 January 13,

30 January 13,

31 January 13,

32 January 13,