Amazon Web Services: The Pla6orm for Data Science. Jamie Kinney

Transcription

1 Amazon Web Services: The Pla6orm for Data Science Jamie Kinney

2 The challenge On premise infrastructure leads to static, lowest common denominator hardware and either long lines for use, low utilization (or both!)

3 What s at stake?

4 Scientific Community Requirements - Computation on demand - a flexible on-demand and cost-effective infrastructure - Data Management - Handling growth of data - Long term storage - Data transfer between participants - Data Analysis - A robust infrastructure - Scalability - Flexibility - Reproducibility of results - A programmable infrastructure

5 AWS Platform Your Applications Management Building Block & Administration Services Identity & Access AWS IAM Identity Federation Consolidated Billing Application Platform Services Content Distribution Amazon CloudFront Foundation Services Compute Amazon EC2 Auto Scale Web Interface Management Console Messaging Amazon SNS Amazon SQS Amazon SES Storage Amazon S3 Amazon EBS Amazon StorageGateway Monitoring Amazon CloudWatch Parallel Processing Elastic MapReduce Database Amazon RDS Amazon SimpleDB Amazon ElastiCache Amazon DynamoDB Deployment & Automation AWS Elastic Beanstalk AWS CloudFormation Simple Workflow Service Libraries & SDKs Java, PHP, Python, Ruby,.NET Networking Amazon VPC Elastic Load Balancing Amazon Route 53 AWS Direct Connect AWS Global Infrastructure Availability Zones Regions Edge Locations

6 AWS Pace of Innovation And wait, there s more! Including: Amazon FPS Red Hat Enterprise on EC Dec 2011 Feb 2012 New Service Announcements & Updates AWS is extraordinarily innovative, exceptionally agile and very responsive to the market. New AWS Region in Sao Paulo, Brazil VPC adds multiple network interfaces Including: Amazon SNS EMR support for cc2.8xlarge Amazon CloudFront S3 announces 48 Object Expiration Amazon Route 53 SNS adds support for Delivery Policies S3 Bucket Policies SNS adds support Including: for Message Formatting RDS Multi-AZ Support Direct Connect Amazon adds RDS four new locations RDS Reserved Databases 24 AWS Free Amazon Usage Tier VPC for Windows AWS Amazon Dynamo EMR DB AWS EC2 IAM Auto Identity Scaling Federation AWS Import/Export AWS IAM Beta AWS Singapore Region EC2 AWS Reserved Storage Instances Gateway Including: Cluster Instances for EC2 AWS EC2 Simple Elastic Workflow Load Balance Service Amazon SimpleDB Micro Instances for EC2 AWS Import/Export Amazon Cloudfront Amazon Linux AMI AWS March Mngmt 2012 Console Amazon EBS Oracle Apps on EC2 New m1.medium Win Srv (22008 ECU s on and EC GB RAM) EC2 Availability Zones SUSE Linux on EC2 Lowered Reserved IBM Apps EC2 on 37% EC2 and RDS 42% EC2 Elastic IP Addresses VM Import for EC ElasicCache in 4 more AWS Regions CloudFront & Route 53 in 3 new edge locations S3 announces Multi-object delete SES now supports SMTP EMR supports Hadoop and Pig Including: AWS Oregon Region Elastic Beanstalk (Beta) Amazon SES (Beta) AWS CloudFormation Amazon RDS for Oracle AWS Direct Connect AWS GovCloud (US) Amazon ElastiCache VPC Virtual Networking VPC Dedicated Instances SMS Text Notification CloudFront Live Streaming AWS Tokyo Region SAP RDS on EC2 SAP BO on EC2 Win Srv 2008 R2 on EC2 Win Srv 2003 VM Import Amazon S3 SSE 2011

7 Global Infrastructure for Global Enterprises GovCloud (US ITAR Region) US West (Northern California) US West (Oregon) US East (Northern Virginia) South America (Sao Paulo) EU (Ireland) Asia Pacific (Singapore) Asia Pacific (Tokyo) AWS Regions AWS Edge Locations

8 AWS Regions and Availability Zones Customer Decides Where Applications and Data Reside

9 Built to Enterprise & Gov t Security Requirements Physical Datacenters in nondescript facilities Physical access strictly controlled Must pass two-factor authentication at least twice for floor access Physical access logged and audited Certifications and Accreditations ISO SSAE 16 / ISAE 3402 / SOC1 (formerly U.S. standard SAS-70 Type II) FISMA Moderate Controls; ITAR region HIPAA applications certified on AWS Payment Card Industry (PCI) Data Security Standard (DSS) Level 1 DIACAP Controls Hardware, Software & Network Systematic change management Phased updates deployment Safe storage decommission Automated monitoring and self-audit Advanced network protection systems Security & Compliance Resources Security & Compliance Center: aws.amazon.com/security Security Overview & Best Practices AWS Risk & Compliance Whitepaper Creating HIPAA Compliant Applications

10 Amazon Simple Storage Service (S3) " Distributed, replicated object store " % durability " ~1 trillion objects and > 700,000 requests/second " Store anything pictures, XML docs, encrypted blobs " You determine the AWS region, we replicate across AZs " AWS Import/Export Service " AWS Storage Gateway

11 Amazon Elastic Compute Cloud (EC2) " Virtual machines running Windows or Linux " Full Windows admin or Linux root privileges " Instance types ranging from t1.micro to cc2.8xl " HPC instances have 10Gb full bisection bandwidth " Ephemeral storage, Elastic Block Storage and SSDs " We constantly modernize our infrastructure

12 Amazon EC2 Pricing Models " On-Demand " Reserved Instances Light Moderate Heavy " Spot " Dedicated Instances

13 Compute: Amazon EC2 Instances

14 2 * Intel Xeon ES-2670 Sandy Bridge Architecture 16 cores w/ HT 60.5 GB RAM 3.4 TB disk HVM cc2.8xlarge

15 2 * 1TB SSD LUNs 16 cores 60.5 GB RAM 35 ECUs 10 Gigabit Ethernet hi1.4xlarge

16 #42

17 Top-5 Pharma Client

18 Not just about high performance infrastructure - Choose the right infrastructure best suited for your applications and pipelines - A programmable infrastructure - No longer bound by physical limits - No queued jobs - Experiment at scale

19 12.7 Teraflops for < $35/hour!

20 Amazon VPC Architecture Customer s isolated AWS resources Subnets Internet NAT VPN Gateway Router Secure VPN Connection over the Internet Amazon Web Services Cloud Customer s Network

21 Database Options Self-Managed Managed Databases Database Server on Amazon EC2 Your choice of database running on Amazon EC2 Bring Your Own License (BYOL) Amazon Relational Database Service (RDS) Oracle or MySQL offered as a service Flexible Licensing: BYOL or License Included Amazon DynamoDB NoSQL data store SSD storage Seamless scalability with zero administration

22 Higher-Level Services Messaging Amazon Simple Queue Service Reliable and highly scalable message queue for cloud applications Amazon Simple Notification Service Push notifications from the cloud to subscribers or client applications Amazon Simple Service Send bulk and transactional s in a quick and costeffective manner Parallel Processing Amazon Elastic MapReduce Allows customers to easily and cost-effectively process vast amounts of data utilizing a Hadoop framework running Amazon EC2 instances Libraries & SDKs Developer Centers Your choice of programming language (Java, PHP, Python, Ruby,.NET) and mobile platform (Android, ios)

23 Deployment & Administration Services Deployment Monitoring Automation AWS CloudFormation Use application templates to create a collection of related AWS in order to provision and update them in an orderly and predictable way Amazon CloudWatch Monitor AWS resources and track metrics to gain insight and react immediately to keep applications running smoothly AWS Elastic Beanstalk Provision an Apache Tomcat environment and deploy your Java applications in minutes

24 Data Collaboration Storage Services Amazon S3 Amazon EBS Amazon DynamoDB Transfer Services AWS Import/Export AWS Storage Gateway Identity and Access Management Federation Encryption features Amazon S3 Server Side Encryption Client side encryption Key Management (Partners)

25 Leverage Public Datasets available on AWS A centralized repository of public data sets Seamless integration with cloud-based applications No charge to the community " Some data sets of interest: Genomes project - Ensembl - Annotated Human Genome Data for FASTA - Illumina - Jay Flateley Human Genome Data Set - YRI Trio Dataset - The Cannabis Sativa Genome - GenBank - UniGene - Influenza Virus - (including updated Swine Flu sequences)

26 AWS Grant Program " " Recipients selected based on: Uniqueness of work Application of Amazon Web Services Ability to disseminate work publicly via papers, events or public relations Great way to develop new public data sets and AMIs

27 Consolidated Billing with IAM " Allows you to get one bill for multiple accounts " You can easily track each account's costs and download the cost data in CSV format " You may be able to reduce costs by combining usage from all the accounts to qualify for volume pricing discounts

28

29

30 Air Force Conducts Large-Scale Aerodynamic Simulation with Amazon EC2 To speed the development of more fuel efficient and durable jet engines, the U.S. Air Force Research Laboratory and AeroDynamic Solutions (ADS) partnered with Amazon Web Services to devise an effective design simulation solution. With Amazon Elastic Compute Cloud (Amazon EC2), ADS proved that large-scale aerodynamic simulations can b dialed up ondemand and performed affordably and within the time constraints of commercial design. Background For the world s leading manufacturers of jet engines, product development remains an extremely costly and time-consuming task. Modern designs have pushed traditional analysis methods to the limit, demanding the use of advanced simulation techniques to better tackle performance and durability issues before committing to hardware. To address these concerns, the Turbine Branch of the United States Air Force Research Laboratory (AFRL) and ADS joined forces to advance one such simulation technique large-scale time accurate simulation for the U.S. gas turbine industry. The Turbine Branch is responsible for advancing the technical capability of turbo propulsion systems, and ADS provides Computational Fluid Dynamics (CFD) software and analysis services to the world s manufacturers of jet engines, industrial gas turbines, and compressors. Time accurate simulation enables designers to understand how time-varying aerodynamic loads can lead to performance loss and structural fatigue. Though this analytical method has long been available, it has largely remained out of reach for commercial design due to its high computational cost and long turnaround time. To carry out large-scale time accurate simulation, hundreds of clustered processors may often be required, necessitating enormous upfront hardware, software and support personnel loaded, transonic, low pressure turbine representative of today's modern designs. For the analysis, one-fifth (1/5) of the full wheel (12/14/7) was simulated for a complete revolution. After generating the mesh and completing initial 3-D multi-stage analysis, the EC2-enabled ADS solution performed as follows: The mesh (10.6 million elements) was partitioned into 40 blocks for parallel execution on the ADS solver Code Leo. 40 processors were dynamically provisioned on Amazon EC2 utilizing five cc1 cluster compute instances. Code Leo was invoked across the 40-processor cluster, simulating 10,500 time steps with 20 inner iterations per time step. Results were gathered and delivered to local servers for post-processing and analysis. EC2 instances were deleted upon completion. Security issues were addressed as well: SAS 70 Type II certification and VPN-level access were required; uploaded and downloaded data was encrypted; dedicated cc1 instances were provisioned to ensure that data mingling did not occur; and data was purged upon completion of the case. Results The results of this case were impressive. Using Amazon EC2 the large-scale, time accurate simulation was turned around in just 72 hours with computing infrastructure costs well below $1,000. Additionally, time accurate analysis revealed critical insights that were not detected using traditional analysis techniques most notably a 2% drop in efficiency relative to conventional 3-D multi-stage steady predictions.

31 Example: Galaxy - An open, web-based platform - Perform, reproduce and share complete analyses - Automatically tracks and manages data provenance and provides support for capturing the context and intent of computational methods

32 Example: Ion Flux - Services to analyze DNA sequence data for researchers and health professionals in genomic medicine

33 For More Information " " " " "

34 Thank you! Jamie Kinney