AWS London Loft: CloudFormation Workshop

Transcription

1 AWS London Loft: CloudFormation Workshop Templated AWS Resources Tom Maddox Solutions Architect

2 Who am I? Gardener (Capacity Planning) Motorcyclist (Agility) Mobile App Writer Problem Solver Technology Geek Solutions Architect

3 What will we cover? Introduction to CloudFormation Template anatomy Bootstrapping instances Managing stacks Best Practices Demo

4 Managing complex cloud architectures

5 Modern cloud architectures have lots of bits

6 Modern cloud architectures have lots of bits EC2 instances Route 53 hosted zones VPCs Security groups S3 buckets IAM users, roles, and policies CloudFront distributions CloudWatch alarms Elastic IPs RDS databases Route tables Auto Scaling groups Elastic Load Balancing SNS topics & subscriptions Elasticache clusters EBS volumes

7 Use the console? EC2 instances VPCs Route 53 hosted zones RDS databases Elastic IPs Security groups S3 buckets Route tables IAM users, roles, and policies CloudFront distributions CloudWatch alarms Auto Scaling groups Elastic Load Balancing SNS topics & subscriptions Elasticache clusters EBS volumes Time consuming Not (easily) auditable Error prone Not repeatable

8 Write a script? EC2 instances VPCs Route 53 hosted zones RDS databases Elastic IPs Security groups S3 buckets Route tables IAM users, roles, and policies CloudFront distributions CloudWatch alarms Auto Scaling groups Elastic Load Balancing SNS topics & subscriptions Elasticache clusters EBS volumes What happens if an API call fails? How can I update my environment? How do I roll back? How long should I pause for?

9 Use CloudFormation?

10 AWS Application Management Services Higher-level services Do it yourself Elastic Beanstalk OpsWorks CloudFormation EC2 Convenience Control

11 CloudFormation Overview Define & manage your infrastructure as code (JSON templates) Declarative approach with managed infrastructure dependencies Self documenting (no more spreadsheets of SG rules) Intelligent updating of resources and automated rollback capabilities AWS API interactions taken care for you Reproducibility Versioning

12 Continuous integration for your complete stack Deploy new template App commit Version Control Pull Jenkins AWS CloudFormation Deploy new app Test Infra commit Amazon S3 Live

13 Continuous integration for your complete stack App commit Version Control Pull Jenkins AWS CloudFormation Promote new template Test Infra commit Amazon S3 Promote new app Live

14 Template anatomy

15 Template structure "Parameters" "Mappings" "Conditions" "Resources" "Outputs"

16 Resources

17 Parameters "Parameters" : { InstanceType : { Description : The EC2 Instance Type to launch., Type : String, AllowedValues : [ t1.micro, m1.small, m1.medium ], InstanceType : { Ref : InstanceType

18 Outputs "Outputs" : { "InstancePublicDnsName" : { "Description" : "The public DNS name of the newly created EC2 instance", "Value" : { Fn::GetAtt" : [ "Ec2Instance, PublicDnsName ]

19 Conditions "Environment" : { "Description" : "Specifies if this a Dev QA or Prod Environment", "Type" : "String", "Default" : "Dev", "AllowedValues" : [ "Dev", "QA", "Prod"], "Conditions" : { "ProdEnvironment" : { "Fn::Equals" : [ { "Ref" : "Environment", "Prod" ], "InstanceType" : { "Fn::If" : [ "ProdEnvironment", m3.2xlarge, m3.medium ]

20 Mappings "Mappings" : { "RegionMap" : { "us-east-1" : { "32" : "ami-6411e20d", "64" : "ami-7a11e213", "us-west-1" : { "32" : "ami-c9c7978c", "64" : "ami-cfc7978a", "eu-west-1" : { "32" : "ami-37c2f643", "64" : "ami-31c2f645", "ap-southeast-1" : { "32" : "ami-66f28c34", "64" : "ami-60f28c32", "ap-northeast-1" : { "32" : "ami-9c03a89d", "64" : "ami-a003a8a1", "ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region", "32"],

21 Helpful Links Search: AWS CloudFormation Templates Helpful solution-based templates (eg: Active Directory domain forest, or LAMP stack) mplatequickref.html Search: AWS CloudFormation Snippets Handy snippets for common AWS services (eg: Autoscaling group) - Search: AWS CloudFormation Template Reference - API docs for each CloudFormation resource type

22 Demo!

23 Bootstrapping instances

24 Use AWS::CloudFormation::Init Declarative Specify the end state, not how to get there Debugable cfn-init.log can integrate into CloudWatch Logs Secure Supports IAM roles Updatable - Supported as part of the stack update process Bring your own tools Use cfn-init to bootstrap Chef, Puppet etc.

25 Use AWS::CloudFormation::Init "Metadata" : { "AWS::CloudFormation::Init" : { "webapp-config": { "packages" : {, "sources" : {, "files" : {, "groups" : {, "users" : {, "commands" : {, "services" : {,

26 Using CloudFormation::Init to update instances 1. Update instance metadata in the template 3. AWS CloudFormation daemon updates configuration "packages" : {, "sources" : {, "files" : {, "groups" : {, "users" : {, "commands" : {, "services" : { 2. UpdateStack Metadata cfn-hup Instance

27 Managing your stacks

28 Organize by layers & environments Frontend Services Backend Services Shared Services Base Network Identity E-Commerce Website Search, Payment Gateway, Reviews, Recommendations Common Monitoring tools, Queues VPCs, Subnets, VPNs, NATs IAM Users, Groups, Roles

29 Apply SOA principles E-Commerce Website Depends On Recommendations Engine "Parameters" : { RecommendationsEndPoint : { Description : URL of the recommendations ELB, Type : String, Wire "Outputs" : { "RecommendationsEndPoint" : { "Description" : "URL of the recommendations ELB", "Value" : { Fn::GetAtt" : [ "RecommendationsELB, PublicDnsName ]

30 Nested stacks for reusability Website1 Resources : { ELB, AutoScaling, RDS Website2 Resources : { ELB, AutoScaling, DynamoDB ELB_AND_AS Resources : { ELB, AutoScaling Website1 Resources : { NestedStack, RDS Website2 Resources : { NestedStack, DynamoDB

31 Deploying Applications

32 Choose a deployment style In-place Blue-Green Templates Stacks Amazon Route 53

33 Use AutoScalingRollingUpdate for zero downtime "UpdatePolicy" : { "AutoScalingRollingUpdate" : { "MaxBatchSize" : 2, "MinInstancesInService" : 2, "PauseTime" : PT20M

34 Use AutoScalingRollingUpdate for zero downtime Developer deploys new template containing change to Launch Configuration CloudFormation replaces a single batch of EC2 instances, the others remain in service Auto Scaling group CloudFormation Stack ELB health check confirms new instances are healthy (otherwise rollback) Next batch of EC2 instances are replaced

35 Optimizing the ASG UpdatePolicy On test environments you can optimize for speed and replace all instances at once Once live, you should update the ASG in batches making sure you don t have downtime

36 for example For a service with an ASG with 6 instances TEST "UpdatePolicy": { "AutoScalingRollingUpdate": { "PauseTime": "PT0S", "MaxBatchSize": 6", "MinInstancesInService": "0" LIVE "UpdatePolicy": { "AutoScalingRollingUpdate": { "PauseTime": "PT15S", "MaxBatchSize": "2", "MinInstancesInService": "2"

37 Best Practices

38 Validate your templates ValidateTemplate API validates: JSON syntax Absence of circular dependencies Template structure

39 Use parameter types "Parameters" : { VpcId" : { "Type" : "AWS::EC2::VPC::Id", SubnetIds" : { "Type" : "List<AWS::EC2::Subnet::Id>", SecurityGroups" : { "Type" : "List<AWS::EC2::SecurityGroup::Id>", KeyPair" : { "Type" : "AWS::EC2::KeyPair::KeyName"

40 Use Deletion Policies to protect resources { "AWSTemplateFormatVersion" : " ", "Resources" : { "mys3bucket" : { "Type" : "AWS::S3::Bucket", "DeletionPolicy" : "Retain" "Properties" : { "BucketName" : MyBucket { "AWSTemplateFormatVersion" : " ", "Resources" : { "myvolume" : { "Type":"AWS::EC2::Volume", "DeletionPolicy" : "Snapshot "Properties" : { "AvailabilityZone" : us-east-1a, "Size : 100 On stack deletion: MyBucket myvolume Snapshot

41 Use stack policies as update guardrails Do not update the databases "Effect" : "Deny", "Principal" : "*", "Action" : "Update:*", "Resource" : "*", "Condition" : { "StringEquals" : { "ResourceType : [ "AWS::RDS::DBInstance, "AWS::Redshift::Cluster ] Okay to update, unless the update requires replacement "Effect" : "Deny", "Principal": "*", "Action" : "Update:Replace", "Resource" : "LogicalResourceId/MyInstance"

42 Stop configuration drift with IAM and tags { "Version": " ", "Statement": [ { "Effect": Deny", "Action": *", "Resource": "*", { "Effect" : Allow", "Action" : [ "Action": "ec2:describe* ], "Condition": { "Null": { "ec2:resourcetag/*cloudformation*" : "true", "Resource" : "*" ]

43 Workshop Challenge

44 Build a web application deployment starting with the VPC Sample Template Sample VPC Template: Master Template: Extra Challenges: AutoScaling Group, ELB & RDS Compatible with any region Fetch Code from Github (passing secrets) Nested Templates for Segregation of Duties A/B Deployment Friendly

45