Amazon Web Services. A Short Guide. SLUUG R. Scott Granneman Jans Carton

Transcription

1 Amazon Web Services A Short Guide SLUUG R. Scott Granneman scott@granneman.com Jans Carton jans@websanity.com 2010 R. Scott Granneman Last updated You are free to use this work, with certain restrictions. For full licensing information, please see the last slide/page. 1

2 2

3 History 3

4 Launched July 2002 Why? Take advantage of underutilized infrastructure, technologies, & operations that power Amazon s $10 billion online store 4

5 60,000 systems 320,000 CPUs 400 Gbps of bandwidth 5

6 June ,000 developers using AWS March ,000,000,000 objects in S3 6

7 Common AWS characteristics Aimed at developers, not end users Billed on usage; no contracts & no minimums 99.95% reliability (some o!er %!) 7

8 Current Services 8

9 AWS Current Services Compute Content Delivery Database E-Commerce Messaging Monitoring Networking Payments & Billing Storage Support Web Tra"c Workforce 9

10 Compute Elastic Compute Cloud (EC2) scalable, pay-as-you-go compute capacity in the cloud Elastic MapReduce easily & cost-e!ectively process vast amounts of data Auto Scaling automatically scale EC2 capacity up or down according to conditions you define 10

11 Content Delivery CloudFront distribute content with low latency via a global network of edge locations 11

12 Database SimpleDB run queries on structured data in real time in conjunction with S3 and EC2 Relational Database Service (RDS) set up, operate, and scale a relational database in the cloud 12

13 E-Commerce Fulfillment Web Service (FWS) allows merchants to deliver products using Amazon.com s worldwide fulfillment capabilities 13

14 Messaging Simple Queue Service (SQS) hosted queue for storing messages as they travel between computers, making it easy to build automated workflow between Web services Simple Notification Service (SNS) set up, operate, and send notifications from the cloud 14

15 Monitoring CloudWatch monitoring for AWS cloud resources, starting with Amazon EC2 15

16 Networking Virtual Private Cloud (VPC) connect existing enterprise infrastructure to isolated AWS compute resources via a VPN Elastic Load Balancing automatically distributes incoming application tra"c across multiple EC2 instances 16

17 Payments & Billing Flexible Payments Service (FPS) facilitates digital transfer of money between any two entities, humans or computers DevPay billing & account management service which enables developers to collect payment for their AWS applications 17

18 Storage Simple Storage Service (S3) fully redundant data storage infrastructure Elastic Block Storage (EBS) block level storage volumes for use with Amazon EC2 instances AWS Import/Export accelerates moving large amounts of data into and out of AWS 18

19 Support AWS Premium Support one-on-one, fast-response support channel 19

20 Web Tra"c Alexa Web Information Service huge repository of data about structure & tra"c patterns on the Web available to developers Alexa Top Sites exposes global website tra"c data as it is continuously collected & updated 20

21 Workforce Amazon Mechanical Turk access thousands of global workers on demand & programmatically integrate their work into various business processes 21

22 S3 22

23 Number of Objects Stored at S3 110,000,000,000 88,000,000,000 66,000,000,000 44,000,000,000 22,000,000,000 0 Oct Jan Oct Mar Mar

24 Who uses S3? SmugMug Dropbox Ubuntu One Slideshare Twitter Woot 24

25 S3: Simple Storage Service Write, read, & delete files Each file can be 1 byte to 5 GB Unlimited number of files Files can be public or private or require authentication Automatic torrents: append?torrent to file s URL Versioning 25

26 Objects are stored in buckets Like folders, but not really Bucket names must be unique across all of S3, not just your account 26

27 Domain mapping Name a bucket test.foo.com In DNS, create a CNAME record pointing test.foo.com to test.foo.com.s3.amazonaws.com. Note: no https access to that bucket without a security alert 27

28 Pricing 28

29 S3 pricing 1 st 50 TB/month Storage Data Transfer In Data Transfer Out PUT, COPY, POST, or LIST requests GET & all other requests $0.15/GB Free til 11/1/10 $0.10/GB $0 for 1 st GB $ /GB $0.01/1000 requests $0.01/10,000 requests 29

30 Full pricing list at aws.amazon.com/s3/#pricing 30

31 Amazon Web Services Simple Monthly Calculator calculator.s3.amazonaws.com/ calc5.html 31

32 32

33 Using S3 33

34 Amazon provides a service, not software 34

35 Many of Amazon s other services use S3 35

36 GUI tools 36

37 Amazon Management Console console.aws.amazon.com/s3/home Web-based Free 37

38 38

39 S3Fox Firefox extension 39

40 DragonDisk Free Windows & Linux 40

41 41

42 Déjà Dup live.gnome.org/dejadup/ Free GNOME 42

43 s3fs code.google.com/p/s3fs/wiki/ FuseOverAmazon FUSE Free 43

44 Cyberduck cyberduck.ch Free Mac OS X 44

45 Transmit $35 Mac OS X 45

46 S3fm Web-based Free or license for integration 46

47 CLI tools 47

48 aws timkay.com/aws/ Perl Free 48

49 JetS3t jets3t.s3.amazonaws.com/index.html Java Free 49

50 s3cmd s3tools.org/s3cmd Python Free Read s3cmd sync HowTo s3tools.org/s3cmd-sync 50

51 S3sync Ruby Free 51

52 Create ~/.s3conf/s3config.yml 52

53 # Your AWS Access Key aws_access_key_id: 0G60EWS0AZ09JTFCB2P2 # Your AWS Secret Key aws_secret_access_key: b9w5ytrebh8jfgtyplzaa0x2lnzx/uohwgrymu+ # Path to Certificate Authority keys ssl_cert_dir: /Users/[username]/bin/s3sync/ s3certs # Number of HTTP errors before s3sync exits s3sync_retries: 5 # Character set s3sync_native_charset: UTF-8 53

54 $ ~/bin/s3sync/s3sync.rb --ssl --delete --verbose --progress --recursive /Users/rsgranne/Documents/Clientele/ granneclientele: Others: --dryrun --debug 54

55 EBS 55

56 EBS: Elastic Block Storage Hard drive storage mounted to EC2 instance 1GB to 1TB Snapshots saved to S3 56

57 EBS pricing $0.10/GB/month $0.10/1 million IO requests 57

58 Public Data Sets on AWS aws.amazon.com/publicdatasets/ repository of public data sets that can be seamlessly integrated into AWS cloud-based applications 58

59 Sample Public Data Sets Annotated Human Genome Data Various US Census Databases UniGene Freebase Data Dump Sloan Digital Sky Survey Wikipedia Extraction 59

60 Managing 60

61 Amazon EC2 API Tools developer.amazonwebservices.com/ connect/entry.jspa?externalid=351 CLI for managing EBS Get info, attach storage, & more 61

62 62

63 EC2 63

64 EC2: Elastic Compute Cloud Servers on the fly, running Linux or Windows Increase or decrease capacity as needed, in minutes Static IP addresses associated with your account, not an instance Elastic Load Balancing Snapshots saved to S3 64

65 Pricing 65

66 EC2 operating systems Red Hat EL Server 2003 OpenSolaris Server 2008 Fedora SUSE Gentoo Amazon Linux Ubuntu Debian 66

67 EC2 instances RAM CPU Storage Bits Small 1.7 GB GB 32 Large 7.5 GB GB 64 Extra Large 15 GB GB 64 High-Memory Quad Extra Large 68 GB GB 64 High-CPU Extra Large 7 GB GB 64 67

68 EC2 instances RAM CPU Storage Bits Micro 613 MB 2 EBS 32/64 68

69 EC2 pricing Small $0.085/hr $0.12/hr Large $0.34/hr $0.48/hr Extra Large $0.68/hr $0.96/hr High-Memory Quad Extra Large $2.00/hr $2.48/hr High-CPU Extra Large $0.68/hr $1.16/hr 69

70 EC2 pricing Micro $0.02/hr $0.03/hr 70

71 Data Transfer EC2 pricing In 1 st 1 GB Out Up to 10 TB Out Next 40 TB Out To & from S3 Free til 11/1/10 $0.10/GB $0 $0.15/GB $0.11/GB $0 71

72 Small EC2 instance $0.085/hr # 24 hours = $2.04/day $14.28/week $63.24/month 72

73 Instances & AMIs 73

74 Uses Xen virtualization A virtual machine is an instance 74

75 You could bootstrap everything Much easier to start with an AMI (Amazon Machine Image) An AMI instantiates an instance 75

76 Amazon AMIs developer.amazonwebservices.com/ connect/kbcategory.jspa? categoryid=171 76

77 77

78 SUSE Linux Enterprise Server 10 Amazon Linux AMI (Cent OS) Gentoo Linux 2010 Ubuntu

79 BitNami LAMPStack BitNami Joomla! Stack BitNami WordPress Stack BitNami Trac Stack BitNami MediaWiki Stack BitNami phpbb Stack BitNami DokuWiki Stack 79

80 IBM WebSphere Application Server IBM Lotus Domino IBM InfoSphere IBM Informix Dynamic Server IBM DB2 Express 80

81 O"cial Ubuntu support for EC2 81

82 O"cial Ubuntu AMIs uec-images.ubuntu.com/ releases/10.04/release/ 82

83 Ubuntu Server on Cloud 10 Program Anyone can try out Ubuntu Server Edition on Amazon EC2 for free for one hour 10.cloud.ubuntu.com 83

84 Alestic alestic.com Ubuntu and Debian AMIs for Amazon EC2 84

85 85

86 86

87 EC2 has no persistent storage Temporary storage survives reboots but not termination Originally relied on S3 for booting Now you can instead use EBS-backed volumes for booting 87

88 If you re starting with an instance-store root device type Create EBS Boot AMI /12/create-ebs-boot-ami.html 88

89 If you re starting with an EBS-backed instance Using Amazon EBS-Backed AMIs and Instances docs.amazonwebservices.com/ AWSEC2/latest/UserGuide/ index.html? Using_BootFromEBS.html 89

90 The basic workflow is 1. Create snapshots of all volumes attached to currently running EBS-backed instance 2. Register an AMI that references those snapshots 90

91 Full set of instructions at files.granneman.com/ presentations/cloud/ HOWTO_Create_EBSbacked_AMI_for_AWS.txt 91

92 Managing 92

93 Amazon EC2 API Tools developer.amazonwebservices.com/ connect/entry.jspa?externalid=351 CLI for managing EC2 Get info, start, terminate, attach storage, associate IP, & more 93

94 94

95 VPC 95

96 VPC: Virtual Private Cloud Connect corporate infrastructure to isolated EC2 instances via VPN 96

97 VPC pricing 1 st 10 TB/month Connection Data Transfer In Data Transfer Out $0.05/hour $0.10/GB $0.15/GB 97

98 Reading 98

99 Je! Bezos Risky Bet Business Week (13 November 2006) content/06_46/b htm 99

100 AWS Documentation aws.amazon.com/documentation/ 100

101 Thank you! Web: Publications: Blog: blog.granneman.com Twitter: scottgranneman 101

102 Amazon Web Services A Short Guide SLUUG R. Scott Granneman scott@granneman.com Jans Carton jans@websanity.com 2010 R. Scott Granneman Last updated You are free to use this work, with certain restrictions. For full licensing information, please see the last slide/page. 102