Cloud Architectures. Jinesh Varia. Amazon Web Services. Technology Evangelist

Transcription

1 Cloud Architectures Jinesh Varia Technology Evangelist Amazon Web Services

2 ANIMOTO.COM

3 Scale: 50 servers to 3500 servers in 3 days

4

5

6 Winner December 2007 Prize:Golden Hammer Photo: Smashing the hardware

7

8

9 TimesMachine from NY Times Articles TIFF -> PDF Input: 11 Million Articles (4TB of data) What did he do? 100 EC2 Instances for 24 hours All data on S3 Output: 1.5 TB of Data Hadoop, itext, JetS3t Under $400

10

11

12

13 CS290F : Scalable Internet Services USCB Fall 2006 Prof created an app to manage team usage Ruby on Rails Complete Stack: From Load balancer, App Server to DB Learn how to scale: Simulated load Generated Graphs All course contents, students assignments, lessons learned are on the Wiki

14 CS345a : Data Stanford Tools used: Shell/Linux/Java Hadoop on EC2 Data set on S3 Datasets :NetFlix, Alexa, IR datasets from TREC Class organization: Stanford Winter Students Each Team spawns Hadoop slave nodes TA created Getting- Started AMIs (& scripts) TA managed the students usage

15

16 Cloud Architectures

17 Cloud Architectures Hardware Infrastructure Job execution time time

18 Cloud Architectures Hardware Infrastructure Job execution time time

19 Shrink your processing time CPUs time

20 Shrink your processing time CPUs time

21 Main Problems Technical How to co-ordinate jobs between machines (distributed processing)? What if a machine fails? How will I Scale-out? Business How do I get management signoff? Resources to manage the infrastructure? How do I get rid of the Idle Infrastructure? Hadoop Web Services Cloud Computing

22 Let s take a usecase Web Company : Analyze large-data sets of clickstream logs Social Networking Company : Analyze demographic and market data Phone Company : Locate all customers who have called in a given area Large Retailer Chain : Wants to know what items a particular customer bought last month Surveillance Company : Wants to transcodevideo for last several years PharmaCompany : Wants locate people who were prescribed a certain drug

23 GrepTheWeb

24 What s so cool about GrepTheWeb? RegEx WWW

25 Examples of Patterns Source Code intx = 40 + i Any thing with punctuation Hey! he said, Are you ok? Case Sensitive Function CallOrderController() Equations f(x) = x^2 Other Patterns (dis)integration of life, Address

26 Zoom Level 1 RegEx Input dataset (List of Document Urls) Alexa GrepTheWeb Service GetStatus Subset of document URLs that matched the RegEx

27 Zoom Level 2 Amazon SQS Distributed Transient Buffer Never Lose a message StartGrep RegEx Amazon SQS Input Files (Alexa Crawl) Amazon S3 Infinitely Scalable Storage in the cloud Ideal for small short-lived Highly Available, Amazon Durable SimpleDB and Reliable Amazon EC2 Database in the cloud messages Manage phases Resizable Computing Private and Public Storage Capacity in the cloud Controller Access control Pay by the GB Lightweight Query-able User info, Attribute Launch, Monitor, Store Spawn Server Instances Job status info Shutdown Message using a Locking Web Service call Root Level Access Amazon SimpleDB DB Amazon EC2 Cluster Amazon S3 GetStatus Get Output Pay by the hour Distributed and Partitioned Input Output Pay by GB, Pay per Query

28 Zoom Level 3 Amazon SQS Billing Queue StartGrep Launch Queue Monitor Queue Shut down Queue Billing Service Controller Launch Monitor Shutdown Controller Controller Controller Billing Controller Insert JobID, Status launch Insert EC2 info ping Get EC2 Info Shutdown Check for results GetStatus Status DB Master M Slaves N HDFS Put File Output Input Get Amazon HadoopCluster on File SimpleDB Amazon EC2 Amazon S3 Get Output Input Files (Alexa Crawl)

29 Zoom Level 4 User1 StartJob 1 Servic e User2 StartJob 2 Map Map Map.. Map Tasks Map Map Map.. Map Tasks Combin e Reduce Hadoop Job Combine Reduce Hadoop Job StopJob 1 StopJob 2 Store status and results Get Result

30 SideTrack: WordCountExample MAPPER: For each input record, extract a set of key/value pairs that we care about the each record Hi Hadoop, Bye Hadoop Input Map Input key value pairs ( Hi, 1), ( Hadoop, 1), ( Bye, 1), ( Hadoop, 1) key 1 Values.. key 3 Values.. REDUCER: For each extracted key/value pair, combine it with other values that share the same key ( Hadoop, [1,1]) Key 1 All Values.. Reduce Aggregate ( Hadoop, 2) Final Key 1 Values.. Source: Doug Cutting s Slide Deck on Hadoop

31 Zoom Level 5 (HadoopMapReduce) MAPPER: For each input record, extract a set of key/value pairs that we care about the each record (LineNumber, s3pointer) Input Map Input key value pairs (s3pointer, [matches]) key 1 Values.. key 3 Values.. REDUCER: For each extracted key/value pair, combine it with other values that share the same key Key 1 All Values.. Reduce Aggregate Identity Function Final Key 1 Values.. Source: Doug Cutting s Slide Deck on Hadoop

32 The Open Source Hadoop framework is giving developers the power to do some pretty extraordinary things.

33 The Open Source Hadoop frameworkon Amazon EC2/S3is giving every developerthe power to do some pretty extraordinary things.

34 References Running Hadoop MapReduce on Amazon EC2 and Amazon S3 D=873 Hadoop on Amazon EC2 Step By Step Wiki Taking Massive Distributed Computing to the Common Man - Hadoop on Amazon EC2/S3

35

36 Thank you! Jinesh Varia /s3 /ec2 /sdb /sqs /forums /resources /blog