CIT 668: System Architecture

Transcription

1 CIT 668: System Architecture Amazon Web Services II Topics 1. Elastic Block Service (EBS) 2. Simple Storage Service (S3) 3. CloudFront 4. CloudWatch (and AutoScaling) 5. Simple Notification Service (SNS) 6. Simple Queue Service (SQS) 7. SimpleDB 8. Relational Database Service (RDS) 9. Route Elastic BeanStalk 11. Flexible Payment Service 12. Mechanical Turk Elastic Block Store (EBS) 1

2 Elastic Block Store (EBS) EBS Volumes are up to 1TB in size Attach to any EC2 instance in same AZ Create snapshots at any time Create new volumes based on snapshots Reliability Annual Failure Rate (AFR) of % Commodity hard disk AFR is ~4% About as reliable as a RAID set Use snapshots for backups Pricing per GB-month EBS Snapshots Snapshots saved to S3 Not visible by S3 API Include only disk blocks changed since last snapshot Snapshots are fast Only changed blocks since last snapshot need to update Creating an EBS Volume Create volume of size 10GB in specified zone ec2addvol -s 10 -z us-east-1c EBS volumes are region specific 2

3 Attaching an EBS Volume Attach volume to an instance ec2attvol -d /dev/sdh -i i-71b6181d vol-9452d0fc Check volume status ec2-describe-volumes vol-9452d0fc Using an EBS Volume Login to instance ssh -i keypath Make partitions fdisk /dev/sdh Create ext3fs filesystem mke2fs j v /dev/sdh1 Mount filesystem mkdir /cit668 mount /dev/sdh1 /cit668 Detaching an EBS Volume Detach volume from an instance ec2-detach-volume vol-9452d0fc Delete a volume permanently ec2-delete-volume vol-9452d0fc Attached volumes cannot be deleted 3

4 Creating an EBS Snapshot ec2addsnap -d 'test snapshot' vol-9452d0fc SNAPSHOT snap-3b0a4156 vol-9452d0fc pending T21:10: test snapshot Attaching an EBS Snapshot Create an EBS volume from the snapshot ec2addvol --snapshot snap-3b0a4156 -z us-east-1c VOLUME vol-aa65e7c2 10 snap-3b0a4156 us-east-1c creating T21:20: Attach the new EBS volume ec2attvol -d /dev/sdi -i i-71b6181d vol-aa65e7c2 Login to instance and mount volume mkdir /cit668-snap mount /dev/sdi1 /cit668-snap Simple Storage Service (S3) 4

5 S3 Features An Internet-scale data storage service All data is stored redundantly in multiple AZs Data is located in the region you specify Stores objects from 1 byte to 5TB in size Objects are stored in a bucket and retrieved via a unique, developer-assigned URL You can have 100 named buckets Each bucket can store an unlimited objects in a flat namespace. S3 Pricing Applications of S3 Fast, scalable, and reliable web file hosting Especially useful for audio and video files 5

6 S3 ACLs Both buckets and objects can have ACLs. ACLs can have up to 100 entries. Owner always has READ_ACP and WRITE_ACP on buckets and objects even if removed in ACL. S3cmd To use for the first time, set up credentials with s3cmd --configure Bucket creation s3cmd mb s3://username-test Upload files s3cmd put www/index.html s3://username-test List files s3cmd ls s3://username-test Download files s3cmd get s3://username-test/index.html /tmp/index.html Remove files s3cmd del s3://username-test/index.html Remove bucket s3cmd mb s3://username-test 6

7 Configure Master Password in Firefox Set Master Password S3 Fox: Manage Accounts 7

8 S3Fox: Create a Bucket 1. Right click in right pane to bring up S3 menu 2. Select Create Directory S3Fox: Uploading Files S3Fox: Edit ACL 8

9 Accessing S3 Files Import/Export 1. Prepare a portable storage device esata or USB external hard drives 2.5 or 3.5 internal SATA hard drives 2. Submit create job request Amazon S3 bucket for store AWS access key ID Return address 3. Securely identify device to placing signature file on it. 4. Ship device will all connectors Import/Export Pricing 9

10 CloudFront CloudFront Amazon CloudFront delivers your static and streaming content using a global network of edge locations. Requests for your objects are automatically routed to the nearest edge location, so content is delivered with the best possible performance. CloudFront = Content Distribution Network Throughput Improvement Instead of being limited to one ISP Use multiple ISPs with much higher bandwidth Ex: If each ISP offers 1 Gbps, then CDN with 10 edge servers at different ISPs offers 10 Gbps Latency Improvement Since edge servers are closer to customers, latency is lower, which is important for streaming media. Availability Since edge servers use different ISPs and power grids, even large scale failures won t bring down entire CDN. 10

11 Other CDNs Free CDNs Commercial CDNs CloudFront Edge Locations US Europe Asia Ashburn, VA Dallas/Fort Worth, TX Jacksonville, FL Los Angeles, CA Miami, FL New York, NY Newark, NJ Palo Alto, CA Seattle, WA St. Louis, MO Amsterdam Dublin Frankfurt London Hong Kong Tokyo Singapore CloudFront Configuration 1. Specify origin server S3 bucket OR Your own server URL 2. Get CloudFront domain: abc.cloudfront.net 3. Use CloudFront domain in your links CloudFront will identify which edge location will best serve your content per 10,000 HTTPS requests (HTTP are 25% less) per GB transferred (lower rates if > 10TB transferred) 11

12 CloudWatch CloudWatch Monitoring service for EC2 CPU utilization Data transfer Storage usage Features Notifications at user-specified metric thresholds Enables AutoScaling at metric thresholds Pricing Basic Monitoring with 5 minute granularity free Detailed Monitoring (1 minute) for 1.5 per hour 10 per alarm after first 10 alarms AutoScaling AutoScaling Group Set of EC2 instances that should scale together Triggers Scale on CloudWatch alerts Scale on time-based schedule Fixed number of healthy instances Examples Add 3 instances if CPU > 50% Remove 3 instances if CPU < 10% 12

13 Simple Notification Service Simple Notification Service (SNS) Setup 1. Create a topic Identifies a specific subject or event type Clients subscribe to topics to get notifications 2. Set topic policies Transport protocols: HTTP, HTTPS, SMTP 3. Add subscribers to topic Subscribers specify end point (URL, address) for notification delivery 4. Publish notifications Topic owners publish update messages to topic SNS Applications Relay events between distributed computing clients Update records in business systems, such as inventory changes and shipping status Relay real time information to mobile devices Monitoring notices from CloudWatch 13

14 Simple Queue Service (SQS) Simple Queue Service (SQS) SQS is a reliable, highly scalable hosted distributed queue for storing messages as they travel between computers. Features Reliable Scalable Secure Used to pipe data between processes (possibly on different instances) Queues Developers can create an unlimited number of queues in each of the 4 regions Messages can be retained for up to 14 days Messages are up to 64KB in size Caveats about distributed message queues Messages may not be delivered in order Messages may be delivered multiple times per 10,000 SQS requests 14

15 SimpleDB SimpleDB Cloud-based non-relational data store Data is stored in domains (tables) Domains have a set of attributes (columns) Attributes can have up to 256 values Domains can have up to a billion items (rows) SimpleDB can be queried using a simple version of SQL Does not support JOIN operations Attributes can be added Dynamically Initial model for person domain Effect of adding Middle name attribute 15

16 SimpleDB Pricing Relational Database Service Relational Database Service Scalable MySQL 5.1 servers in the cloud with Automated backups and snapshots Replication (across availability zones if needed) RDS instance classes 16

17 Route 53 Route 53 Cloud-based DNS hosting Must have your own DNS domain Pricing per hosted zone per month per million DNS queries per month (up to 1 billion) per million DNS queries per month (after 1 billion) Elastic Beanstalk 17

18 Elastic Beanstalk PaaS with many platforms, configurable IaaS To deploy Java applications using Elastic Beanstalk: 1. Create your application as you normally would 2. Package your deployable code into a standard Java Web Application Archive (WAR file). 3. Upload your WAR file to Elastic Beanstalk 4. Deploy your application. Elastic Beanstalk handles provisioning of a load balancer deployment of your WAR file to EC2 instances running Apache Tomcat application server. Access application via customized URL An Elastic Beanstalk Application Configuration Server - EC2 instance type, security groups, key pair (for SSH login), CloudWatch monitoring interval, and AMI ID (for customization). Load Balancer - SSL Certificate, health check intervals, URL Auto Scaling - Minimum and maximum instance count, and nine settings for the trigger that is used to drive scaling decisions. Notifications - An address that can receive notification of important application events such as scaleup and scale-down actions initiated by Auto Scaling. Container - With the default (Java / Tomcat 6) container, this tab lets you control the size of the Java Virtual Machine's heap and garbage collection parameters. 18

19 Flexible Payments Service Flexible Payments Service (FPS) Allows you to use Amazon s payment service Customers must have an Amazon.com account Can also be used to pay third parties Features One-time payments Configure periodic and delayed payments Aggregate micropayments into single transaction Facilitate transactions between buyer and a third party seller, while taking a cut of the fee Mechanical Turk 19

20 Mechanical Turk How It Worked How It Works on Amazon Requester (Developer) Human Intelligence Tasks (HITs) Worker Qualifications Completed HITs Workers 20

21 Applications Software Developers Businesses Anyone Solves problem of building applications that until now have not worked well without human intelligence Efficiently, effectively and inexpensively complete millions of tasks that require human intelligence 1. Make money using the skills you already have whenever your schedule allows. 2. Previously unimaginable business ideas are now feasible Business Model Podcast transcription service provider Transcribes audio into high-quality text Amazon Mechanical Turk: Workers transcribe podcasts and index text within search engine Amazon S3: Stores podcasts and related files Key Points Simple Storage Service (S3) is scalable web based storage based on Buckets that contain a variable number of objects Objects in buckets accessible via S3 URL A Content Distribution Network (CDN) replicates and distributes content through a network of edge nodes near the customers. Queue services: Simple Notification Service for real-time notifications Simple Queue Service for piping messages between app components Cloud services: SimpleDB is non-relational datastore with simple SQL RDS uses configurable MySQL 5.1 instances Route 53 provides scalable DNS Mechanical Turk leverages global workforce to perform small tasks for micropayments 21