Scaling the Cloud Kirk Spadt Principal Architect Automated Financial Systems January 26, 2010 King of Prussia, PA Philadelphia Java Users Group
Next big thing? Sharing large server computers Information System Utility - like water, electric, phone Old technology 1970's: "mainframe", "time-sharing", "virtual machine 1980's: "network", "LAN", "WAN", "Internet" 1990's: "World-Wide Web", "hosting service", "groupware", "ASP" 2000's: Virtualization on Intel CPUs (VMWare, Xen, Parallels) Mix them together, plus a jazzy new name Technology is already mature Why now? Broadband! Everyone has it, and it's not expensive Distance to data the CPU running the app is irrelevant Slide content courtesy of Fred Stluka
The tipping point Pressures: The economy Better Faster Cheaper Globalization Time to Market Outsourcing New generation Google Enablers: World Wide Web Broadband Web 2.0 New generation Social networking Network effect Open source
Cloud Computing SAAS: Stack Software... PAAS: Platform... IAAS: Infrastructure...... as a service http://en.wikipedia.org/wiki/cloud_computing
SAAS: Software as a service Gmail, Facebook, YouTube, Twitter, Skype Quicken Online, Blogs, Wikis, Online backup SalesForce, Run-It Retail, Security scans Characteristics Subscription-based Available anywhere No upgrades Less virus risk Can collaborate Harder to lose data Who? Everyone. No technical skills Not customizable Social interaction Users not programmers We re not needed
PAAS: Platform as a service Google App Engine, Force.com, GigaSpaces, LongJump CRM, NetSuite, WorkXpress 5GL Microsoft Azure, Oracle PAAS, Open Platform Characteristics Function-based No upgrades Programming API You create the apps Default applications No root access Vendor lock-in? Who? Must have technical skills. Can be customized No infrastructure worries No system upgrades Scaling handled for you We create + customize We share costs and risks
IAAS: Infrastructure as a service Amazon Elastic Compute Cloud, Joyent (Facebook s) IBM Blue Cloud, CloudWare, RackSpace VCloud (VMware), GoGrid, AT&T Cloud Services Characteristics Usage-based Root access No upgrades No program apps No platform APIs Programming API You create the apps Who? Hardware failures handled No infrastructure worries We do system upgrades We handle the scaling We can add value (profit) Less vendor lock-in More profit and more risk
Why Amazon Cloud? Enabling technologies are compelling More control of the environment Open source provides many solutions Cost has triggered a tipping point IAAS utilizes more of our skills Less vendor lock-in than PAAS/SAAS Lower cost, higher potential profit/gain
Key Amazon AWS Components S3 Simple Storage Service Provides public/private access to your data EC2 Elastic Compute Cloud Linux or Windows server instances (as root) RDS Relational Database Service A managed and replicated MySql database SQS Simple Queue Service Guaranteed single-delivery message queues
EC2 (Server) Components Wide range of CPU power priced accordingly Can add/delete servers on demand billed hourly Preconfigured server images with applications Windows and multiple Linux distributions Some have preloaded apps (e.g., Apache, Tomcat, MySql) Some are paid/licensed, (e.g., WindowsServer. DB2) Elastic Block Storage (EBS) reliable persistent store Data survives server shutdown (otherwise it s gone) Serves as a Unix mount point or D: drive Elastic IP addresses fixed; can swap among servers Amazon CloudWatch resource utilization metrics + reports Auto Scaling uses metrics to add instances as needed Responds to CloudWatch metrics and scales up and down Elastic Load Balancing Distributes requests Responds to health of servers, routes accordingly
Sample Amazon AWS Pricing Small: 1cpu, 32bit, 1.7G ram: 8.5 / hr Large: 2cpu, 64bit, 7.5G ram; 34 / hr Up to 4 fast cpus and 68GB memory available ($2.40/hr) Linux pricing is above Windows is 20-30% higher Reserved instances (e.g. small is 3 / hr + $233 / year) Spot price instances only run if below a price you set. Data transfer: from outside: 17 / GB; in region: 1 / GB EBS storage volumes: 10 / GB / month CloudWatch monitoring 1.5 / server / hour Load balancing: 2.5 / hour + 0.8 / GB passed through Net: ~ $70/month for a small or $32-42/month if reserved. Note: Prices are as of 01/26/2010; subject to change.
Amazon Cloud Demo Creating/storing a key pair for secure access Defining the firewall (accessible ports) Selecting an AMI (Amazon machine image) Starting an EC2 (server) instance These slides are intended for those of you who are not here tonight.
Prerequisites for the demo An Amazon account tied to an active credit card This is the same account as for books. An SSH client for secure terminal access PuTTY - the standard for most of us, and free. http://www.chiark.greenend.org.uk/~sgtatham/putty/ A SCP client for secure file transfer WinSCP well-respected and also fee http://winscp.net/eng/download.php Signed up for an Amazon AWS account. [Sign up now] at http://aws.amazon.com/
Amazon Cloud Demo Links http://aws.amazon.com/ The sign up link for an AWS account https://console.aws.amazon.com/ec2/home The EC2 administrative console http://aws.amazon.com/documentation/ AWS documentation http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryid=172 Tutorials http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryid=171 Machine image directory (OS, preloaded apps, etc).
Amazon AWS Management Console
AWS Console Details
Adding a key pair
About AWS Key Pairs Key pairs are used instead of passwords. Better security, control, easier login. You must have a key pair to start a server. The private key is saved in Putty (SSH). Private key is downloaded upon creation. Use separate keypairs by client. DO NOT lose your private key.
Updating the default security group
About AWS Security Groups Multiple groups can be assigned (per server) I assign an ssh group (just port 22 in it) I restrict access to my ip(s) so easily changed. I assign a security group per server role. apache has port 80; mysql has no ports This technique also documents the servers. I assign a security group per server farm I open up all ports between these servers That is why I do not need to open 3306 for mysql
AMI (machine image) selection
About AMI Selection The AMI you select controls what is on the server Operating system (Linux flavor or Windows) Preinstalled applications and tools Many AMIs are available for Linux (servers also cheaper) It is easy to add apps to Linux images yum install httpd installs Apache in one step (then you configure) I started with RightScale CentOS-5.4 i386 This is a clone of RedHat Enterprise Server 5.4 It contains Apache, MySql, Tomcat Ubuntu and Fedora AMIs are more current Also have more issues based on the chatter I hear Most AMIs are free even from vendors But, you ll pay Oracle, WebSphere, per hour via Amazon
About the Running Instances Wizard Choose the availability zone carefully Same farm in different zones incurs transfer cost However, true redundancy requires multiple zones. If you have a reserved instance, use its zone. Turn monitoring ON if balancing or auto-scaling Be sure you have the private key for the key pair. I select a group for ssh, each role, and the farm Foo.com apache would have groups foo, apache, ssh Farms need a group security group with more ports open Consider a separate ssh group to limit access by ip addr Do not use this wizard if you are auto-scaling. This information is specified in an auto-scale launch configuration
Scaling? Handle increases in load improve performance. Handle decreases in load reduce costs. Adjust number of app servers based on demand Adjust database server count Assure that user sessions are preserved Assure stored data integrity My costs should reflect my current needs. My systems should respond to demand as needed
Server Farm Topology MySql Master-Slave Client Requests Web Load Balancer www.foo.com Port 80 DNS points to (example): Lb1234.elb.amazon.com Port 80 Forwards to port 8080 Tomcat Tomcat 8080 Tomcat 8080 Tomcat 8080 8080 Terracotta client does balancing queries transactional jdbc driver selects Terracotta Terracotta Server Server MySql Master SlaveDb Load Balancer MySql MySql Slave MySql Slave MySql Slave MySql Slave Slave
Terracotta - Functions A powerful distributed Java object cache No serialization java classes are instrumented instead Concurrency leverages synchronized { } blocks Shares heap memory among all participants You can configure (in xml) what objects are shared Deployed with 1 or more servers that coordinate state Effective heap size for each java client is the sum of all High performance: classes communicate each field change Custom TIM (integration modules) for: Hibernate second-level cache Session persistence for many app servers Spring integration failover supports all webflow scopes Use these roll your own only if you are a concurrency expert
AWS Load Balancing AWS provides a load balancing solution Define an Elastic Load Balancer Specify its listening and forwarding ports For Tomcat: lb=80; instance=8080 For MySql: lb=3306; instance=3306 AWS generates a host URL for the balancer Register this in the website s DNS as a CNAME www.foo.com now forwards to the load balancer The load balancer knows the ip of each Tomcat
AWS Auto Scaling Create a load balancer, as described above Create a launch configuration References the load balancer Contains the specs needed to launch servers Create an auto-scale group References the launch configuration Specifies the min and max deployed servers Create an auto-scale trigger (or more than one) References the auto-scale group Specifies what metrics to observe and limits CloudWatch (monitoring) must be enabled. Auto scaling and load balancing respond to the metrics.
Command-Line AWS Tools Some AWS functions are not in the console For these functions, use the command line The following slides utilize the CLI The ^s at the end are for the Windows CLI
Load Balancing Configuration elb-create-lb blurt-load-tomcat ^ --availability-zones us-east-1a ^ --listener "protocol=http,lb-port=80,instance-port=8080" blurt-db-load: your name for the load balancer us-east-1a: preferred server availability zone protocol-http: Protocol of the content being forwarded lb-port: Listening port (external - used by requests) instance-port: Forwarding port (internal - to app servers) Responds with a DNS name for the load balancer: blurt-load-tomcat-1790308710.us-east-1.elb.amazonaws.com
Auto Scaling Configuration as-create-launch-config blurt-config-tomcat ^ --image-id ami-33ab475a ^ --instance-type m1.small ^ --key blurt ^ --group blurt,tomcat,ssh ^ --user-data "descr=blurt MySql;cfg=blurt-001;roles=mysql blurt-config-tomcat: your name for the launch config ami-33ab475a: Amazon machine image to be launched m1.small: Architecture and size of the hardware blurt: key pair to use for authentication blurt tomcat ssh: Security group(s) controlling access to the server user-data: A string to use at start-up to configure the server.
Auto Scaling Group as-create-auto-scaling-group blurt-group-tomcat ^ --launch-configuration blurt-config-tomcat ^ --availability-zones us-east-1a ^ --min-size 1 --max-size 1 ^ --cooldown 120 ^ --load-balancers blurt-load-tomcat blurt-group-tomcat: your name for the scaling group blurt-config-tomcat: name you used for the launch configuration us-east-1a: list of availability zones in which to launch servers min-size: Minimum number of servers to launch and maintain max-size: Maximum number of servers to launch cooldown: Number of seconds to wait between scaling changes
Auto Scaling Trigger as-create-or-update-trigger blurt-trig-mysql-cpu ^ --auto-scaling-group blurt-group-mysql ^ --namespace "AWS/EC2" ^ --measure CPUUtilization ^ --statistic Average ^ --dimensions "AutoScalingGroupName=blurt-group-mysql" ^ --period 60 ^ --lower-threshold 20 ^ --upper-threshold 70 ^ "--lower-breach-increment=-1" ^ --upper-breach-increment 1 ^ --breach-duration 120 Defines criteria for all servers in an auto scaling group Measures average CPU utilization over 60 seconds (for the entire group) Adds 1 server if utilization exceeds 70 percent Removes 1 server if utilization goes below 20 percent
Configuration: blurt-001.cfg mysql.dbname=blurt mysql.dbuser=blurt mysql.dbpass=11111111 mysql.master.host=dbmaster.blurt.com mysql.slave.host=lb-1234.elb.amazon.com mysql.initd=mysqld tomcat.hostname=www.blurtthat.com tomcat.war=root:blurt-0.1.war tomcat.initd=tomcat Contains settings for each server role Server information, settings, credentials, hosts, defaults Initialization scripts selectively apply settings based on roles Server applies this configuration at first boot-up Eliminates need for unique AMI images for each server role
Server Farm MySql Master-Slave Issues Only the master accepts write requests Suitable for many reads, few writes Slaves refresh asynchronously - not ACID All transactional requests must be on the master For applications that tolerate stale reads App is responsible for routing to master/slave DB abstraction layers or JDBC drivers are often used for routing Load balancer can be used to route to slaves (not master) ACID reads must route to master a challenge Some shops address this using @Transactional annotations Failover/high availability solutions are a compromise Slave failover is easy load balancer can handle it. Master failover: read-lock slave, wait until sync, switch to master. This requires a controller server or a smart JDBC driver. For higher availability or scalability than this, use MySql Cluster.
Alternative MySql Solutions MySql replication JDBC driver (mysql.com) driver.connect("jdbc:mysql://m1,s1,s2,s3/db ); Routes writes to master and load-balances reads. lbpool connection pool (tailrank.com) JDBC pool with heartbeat runtime load balancing MySql Clustering (NDB) server engine (mysql.com) An outgrowth of an in-memory database Huge memory requirements May need multiple clusters for failover or to scale up MySQL Enterprise for Amazon EC2 (mysql.com) MySql s fee-based solution for EC2, includes failover Amazon EC2 MySql RDS (aws.amazon.com) MySql managed relational database There is no mention of intrinsic failover. Amazon s value-add is backup and patches for now anyway
Investigations and Future Research CloudTools: Open source AWS deployment Basis for Spring Source s CloudFoundry No auto scaling http://code.google.com/p/cloudtools/ Scalr: Open source auto scaling console Similar function to RightScale AWS monitoring service Requires (builds) a unique AMI for each server role in each farm https://scalr.net/login.php AppScale: Open source Google App Engine Can be deployed into Amazon Limited function for now http://code.google.com/p/appscale/ Eucalyptus: Open source Xen virtualization Open source implementation of Amazon AWS same interfaces Looks like a promising way to move servers in-house http://www.eucalyptus.com/
Thank you for your kind attention.