Amazon Web Services A Short Guide SLUUG R. Scott Granneman scott@granneman.com www.granneman.com Jans Carton jans@websanity.com 2010 R. Scott Granneman Last updated 20101013 You are free to use this work, with certain restrictions. For full licensing information, please see the last slide/page. 1
2
History 3
Launched July 2002 Why? Take advantage of underutilized infrastructure, technologies, & operations that power Amazon s $10 billion online store 4
60,000 systems 320,000 CPUs 400 Gbps of bandwidth 5
June 2007 330,000 developers using AWS March 2010 102,000,000,000 objects in S3 6
Common AWS characteristics Aimed at developers, not end users Billed on usage; no contracts & no minimums 99.95% reliability (some o!er 99.999999999%!) 7
Current Services 8
AWS Current Services http://aws.amazon.com/products/ Compute Content Delivery Database E-Commerce Messaging Monitoring Networking Payments & Billing Storage Support Web Tra"c Workforce 9
Compute Elastic Compute Cloud (EC2) scalable, pay-as-you-go compute capacity in the cloud Elastic MapReduce easily & cost-e!ectively process vast amounts of data Auto Scaling automatically scale EC2 capacity up or down according to conditions you define 10
Content Delivery CloudFront distribute content with low latency via a global network of edge locations 11
Database SimpleDB run queries on structured data in real time in conjunction with S3 and EC2 Relational Database Service (RDS) set up, operate, and scale a relational database in the cloud 12
E-Commerce Fulfillment Web Service (FWS) allows merchants to deliver products using Amazon.com s worldwide fulfillment capabilities 13
Messaging Simple Queue Service (SQS) hosted queue for storing messages as they travel between computers, making it easy to build automated workflow between Web services Simple Notification Service (SNS) set up, operate, and send notifications from the cloud 14
Monitoring CloudWatch monitoring for AWS cloud resources, starting with Amazon EC2 15
Networking Virtual Private Cloud (VPC) connect existing enterprise infrastructure to isolated AWS compute resources via a VPN Elastic Load Balancing automatically distributes incoming application tra"c across multiple EC2 instances 16
Payments & Billing Flexible Payments Service (FPS) facilitates digital transfer of money between any two entities, humans or computers DevPay billing & account management service which enables developers to collect payment for their AWS applications 17
Storage Simple Storage Service (S3) fully redundant data storage infrastructure Elastic Block Storage (EBS) block level storage volumes for use with Amazon EC2 instances AWS Import/Export accelerates moving large amounts of data into and out of AWS 18
Support AWS Premium Support one-on-one, fast-response support channel 19
Web Tra"c Alexa Web Information Service huge repository of data about structure & tra"c patterns on the Web available to developers Alexa Top Sites exposes global website tra"c data as it is continuously collected & updated 20
Workforce Amazon Mechanical Turk access thousands of global workers on demand & programmatically integrate their work into various business processes 21
S3 22
Number of Objects Stored at S3 110,000,000,000 88,000,000,000 66,000,000,000 44,000,000,000 22,000,000,000 0 Oct. 2007 Jan. 2008 Oct. 2008 Mar. 2009 Mar. 2010 23
Who uses S3? SmugMug Dropbox Ubuntu One Slideshare Twitter Woot 24
S3: Simple Storage Service Write, read, & delete files Each file can be 1 byte to 5 GB Unlimited number of files Files can be public or private or require authentication Automatic torrents: append?torrent to file s URL Versioning 25
Objects are stored in buckets Like folders, but not really Bucket names must be unique across all of S3, not just your account 26
Domain mapping Name a bucket test.foo.com In DNS, create a CNAME record pointing test.foo.com to test.foo.com.s3.amazonaws.com. Note: no https access to that bucket without a security alert 27
Pricing 28
S3 pricing 1 st 50 TB/month Storage Data Transfer In Data Transfer Out PUT, COPY, POST, or LIST requests GET & all other requests $0.15/GB Free til 11/1/10 $0.10/GB $0 for 1 st GB $0.11 0.15/GB $0.01/1000 requests $0.01/10,000 requests 29
Full pricing list at aws.amazon.com/s3/#pricing 30
Amazon Web Services Simple Monthly Calculator calculator.s3.amazonaws.com/ calc5.html 31
32
Using S3 33
Amazon provides a service, not software 34
Many of Amazon s other services use S3 35
GUI tools 36
Amazon Management Console console.aws.amazon.com/s3/home Web-based Free 37
38
S3Fox www.s3fox.net Firefox extension 39
DragonDisk www.dragondisk.com Free Windows & Linux 40
41
Déjà Dup live.gnome.org/dejadup/ Free GNOME 42
s3fs code.google.com/p/s3fs/wiki/ FuseOverAmazon FUSE Free 43
Cyberduck cyberduck.ch Free Mac OS X 44
Transmit www.panic.com/transmit/ $35 Mac OS X 45
S3fm www.s3fm.com Web-based Free or license for integration 46
CLI tools 47
aws timkay.com/aws/ Perl Free 48
JetS3t jets3t.s3.amazonaws.com/index.html Java Free 49
s3cmd s3tools.org/s3cmd Python Free Read s3cmd sync HowTo s3tools.org/s3cmd-sync 50
S3sync www.s3sync.net Ruby Free 51
Create ~/.s3conf/s3config.yml 52
# Your AWS Access Key aws_access_key_id: 0G60EWS0AZ09JTFCB2P2 # Your AWS Secret Key aws_secret_access_key: b9w5ytrebh8jfgtyplzaa0x2lnzx/uohwgrymu+ # Path to Certificate Authority keys ssl_cert_dir: /Users/[username]/bin/s3sync/ s3certs # Number of HTTP errors before s3sync exits s3sync_retries: 5 # Character set s3sync_native_charset: UTF-8 53
$ ~/bin/s3sync/s3sync.rb --ssl --delete --verbose --progress --recursive /Users/rsgranne/Documents/Clientele/ granneclientele: Others: --dryrun --debug 54
EBS 55
EBS: Elastic Block Storage Hard drive storage mounted to EC2 instance 1GB to 1TB Snapshots saved to S3 56
EBS pricing $0.10/GB/month $0.10/1 million IO requests 57
Public Data Sets on AWS aws.amazon.com/publicdatasets/ repository of public data sets that can be seamlessly integrated into AWS cloud-based applications 58
Sample Public Data Sets Annotated Human Genome Data Various US Census Databases UniGene Freebase Data Dump Sloan Digital Sky Survey Wikipedia Extraction 59
Managing 60
Amazon EC2 API Tools developer.amazonwebservices.com/ connect/entry.jspa?externalid=351 CLI for managing EBS Get info, attach storage, & more 61
62
EC2 63
EC2: Elastic Compute Cloud Servers on the fly, running Linux or Windows Increase or decrease capacity as needed, in minutes Static IP addresses associated with your account, not an instance Elastic Load Balancing Snapshots saved to S3 64
Pricing 65
EC2 operating systems Red Hat EL Server 2003 OpenSolaris Server 2008 Fedora SUSE Gentoo Amazon Linux Ubuntu Debian 66
EC2 instances RAM CPU Storage Bits Small 1.7 GB 1 160 GB 32 Large 7.5 GB 4 850 GB 64 Extra Large 15 GB 8 1690 GB 64 High-Memory Quad Extra Large 68 GB 26 850 GB 64 High-CPU Extra Large 7 GB 20 1690 GB 64 67
EC2 instances RAM CPU Storage Bits Micro 613 MB 2 EBS 32/64 68
EC2 pricing Small $0.085/hr $0.12/hr Large $0.34/hr $0.48/hr Extra Large $0.68/hr $0.96/hr High-Memory Quad Extra Large $2.00/hr $2.48/hr High-CPU Extra Large $0.68/hr $1.16/hr 69
EC2 pricing Micro $0.02/hr $0.03/hr 70
Data Transfer EC2 pricing In 1 st 1 GB Out Up to 10 TB Out Next 40 TB Out To & from S3 Free til 11/1/10 $0.10/GB $0 $0.15/GB $0.11/GB $0 71
Small EC2 instance $0.085/hr # 24 hours = $2.04/day $14.28/week $63.24/month 72
Instances & AMIs 73
Uses Xen virtualization A virtual machine is an instance 74
You could bootstrap everything Much easier to start with an AMI (Amazon Machine Image) An AMI instantiates an instance 75
Amazon AMIs developer.amazonwebservices.com/ connect/kbcategory.jspa? categoryid=171 76
77
SUSE Linux Enterprise Server 10 Amazon Linux AMI (Cent OS) Gentoo Linux 2010 Ubuntu 10.04 78
BitNami LAMPStack BitNami Joomla! Stack BitNami WordPress Stack BitNami Trac Stack BitNami MediaWiki Stack BitNami phpbb Stack BitNami DokuWiki Stack 79
IBM WebSphere Application Server IBM Lotus Domino IBM InfoSphere IBM Informix Dynamic Server IBM DB2 Express 80
O"cial Ubuntu support for EC2 81
O"cial Ubuntu AMIs uec-images.ubuntu.com/ releases/10.04/release/ 82
Ubuntu Server on Cloud 10 Program Anyone can try out Ubuntu 10.10 Server Edition on Amazon EC2 for free for one hour 10.cloud.ubuntu.com 83
Alestic alestic.com Ubuntu and Debian AMIs for Amazon EC2 84
85
86
EC2 has no persistent storage Temporary storage survives reboots but not termination Originally relied on S3 for booting Now you can instead use EBS-backed volumes for booting 87
If you re starting with an instance-store root device type Create EBS Boot AMI www.capsunlock.net/ 2009/12/create-ebs-boot-ami.html 88
If you re starting with an EBS-backed instance Using Amazon EBS-Backed AMIs and Instances docs.amazonwebservices.com/ AWSEC2/latest/UserGuide/ index.html? Using_BootFromEBS.html 89
The basic workflow is 1. Create snapshots of all volumes attached to currently running EBS-backed instance 2. Register an AMI that references those snapshots 90
Full set of instructions at files.granneman.com/ presentations/cloud/ HOWTO_Create_EBSbacked_AMI_for_AWS.txt 91
Managing 92
Amazon EC2 API Tools developer.amazonwebservices.com/ connect/entry.jspa?externalid=351 CLI for managing EC2 Get info, start, terminate, attach storage, associate IP, & more 93
94
VPC 95
VPC: Virtual Private Cloud Connect corporate infrastructure to isolated EC2 instances via VPN 96
VPC pricing 1 st 10 TB/month Connection Data Transfer In Data Transfer Out $0.05/hour $0.10/GB $0.15/GB 97
Reading 98
Je! Bezos Risky Bet Business Week (13 November 2006) www.businessweek.com/magazine/ content/06_46/b4009001.htm 99
AWS Documentation aws.amazon.com/documentation/ 100
Thank you! Email: scott@granneman.com Web: www.granneman.com Publications: www.granneman.com/pubs Blog: blog.granneman.com Twitter: scottgranneman 101
Amazon Web Services A Short Guide SLUUG R. Scott Granneman scott@granneman.com www.granneman.com Jans Carton jans@websanity.com 2010 R. Scott Granneman Last updated 20101013 You are free to use this work, with certain restrictions. For full licensing information, please see the last slide/page. 102