Accelerating Software Delivery with the AWS Developer Tools. Pradyumna Dash Solutions Architect, UK Public Sector

Transcription

1 Accelerating Software Delivery with the AWS Developer Tools Pradyumna Dash Solutions Architect, UK Public Sector

2 Accelerating Software Delivery with the AWS Developer Tools Pradyumna Dash Solutions Architect, UK Public Sector

3 the online book and decorative pillow seller Amazon.com swooped in and, in 2006, launched its own computer rental system the future Amazon Web Services. The once-fledgling service has since turned cloud computing into a mainstream phenomenon Source: Bloomberg Business - April 22, 2015

4

5 Experience Building and managing cloud since 2006 Service Breadth & Depth 90+ services to support any cloud workload Global Footprint 16 regions, 42 availability zones, 73 edge locations Pricing Philosophy 59 proactive price reductions to date Ecosystem Thousands of partners;; Marketplace products

6 16 Regions 73 Edge Locations Region & Number of Availability Zones 42 AZs New Region (coming soon)

7 Sample EU Region Sample Availability Zone Availability Zone A Availability Zone B Datacenter Datacenter Availability Zone C Datacenter

8 The AWS Platform Marketplace Business Applications DevOps Tools Business Intelligence Mgmt. Tools Monitoring Auditing Service Catalog Analytics Query Large Data Sets Elasticsearch Business Analytics Dev Tools Artificial Intelligence IoT Mobile Enterprise Applications Game Development Account Support Security Server Management Hadoop/Spark Rules Engine Build, Test, Monitor Apps Document Sharing Support Networking Configuration Tracking Real-time Data Streaming Private Git Repositories Voice & Text Chatbots Local Compute and Sync Push Notifications & Calendaring Managed Services Database & Storage Optimization Orchestration Workflows Continuous Delivery Machine Learning Device Shadows Build, Deploy, Manage APIs Hosted Desktops Professional Services SaaS Subscriptions Resource Templates Managed Search Build, Test, and Debug Text-to-Speech Device Gateway Device Testing Application Streaming 3D Game Engine Partner Ecosystem Operating Systems Automation Managed ETL Deployment Image Analysis Registry Identity Backup Multi-player Backends Training & Certification Solution Architects Account Management Security & Pricing Reports Technical Acct. Management Migration Hybrid Application Services Security Database Application Discovery Data Integration Application Migration Integrated Networking Data Migration Identity Federation Transcoding Step Functions Messaging Identity & Access Key Storage & Management Active Directory Database Migration Resource Management DDoS Protection Server Migration VMware on AWS Application Analysis Aurora MySQL PostgreSQL Oracle SQL Server Devices & Edge Systems Certificate Management MariaDB Web App. Firewall Data Warehousing NoSQL Exabyte-scale Storage Object Storage Archive Block Storage Data Transport Managed File Storage Compute Virtual Machines Web Simple Servers Auto Scaling Batch Applications Containers Event-driven Computing Networking Isolated Resources Dedicated Connections Global CDN Load Balancing Scalable DNS Infrastructure Regions Availability Zones Points of Presence

9 Why are we here today?

10 Software moves faster today Software creation and distribution is easier and faster than ever: Startups can now take on giants with little to no funding ahead of time Getting your software into the hands of millions is a download away Your ability to move fast is paramount to your ability to fight off disruption

11 The software delivery model has drastically changed Old software delivery model New software delivery model

12 What tools do you need to move fast? Releasing software in this new software-driven world requires a number of tools: Tools to manage the flow of your software development release process Tools to properly test and inspect your code for defects and potential issues Tools to deploy your applications

13 First, we need to understand a little bit about software release processes

14 Release processes have four major phases Source Build Test Production Check-in source code such as.java files. Peer review new code Compile code Unit tests Style checkers Code metrics Create container images Integration tests with other systems Load testing UI tests Penetration testing Deployment to production environments

15 Release processes levels Source Build Test Production Continuous integration Continuous delivery Continuous deployment

16 microservices + 2 pizza teams

17 We move pretty fast at Amazon: In 2014: Thousands of service teams across Amazon + Building microservices + Practicing continuous delivery + Many environments (staging, beta, production, multiple regions) =50 million deploys

18 DevOps: Culture + Practices + Tools Each 2-pizza team owns their product: Creates product (software typically) Handles Q/A of that product Responds to issues, is on-call you build it, you run it Supports service & tracks/goals against business and technical metrics

19 DevOps: Culture + Practices + Tools Each 2-pizza team s practices largely open so far as standards are met: Agile? Scrum? Daily standups? Weekly? None? Whatever you works for your team! No centralized change management board/team/approval, but tools that require a degree of signoff/process review

20 DevOps: Culture + Practices + Tools Each 2-pizza team developers given a box of tools : Use these or operationalize your own: Time spent on operations is less time to spend on development Less time spent on development is increased risk of missing goals Tools provide guard rails that enforce best/better practices Tools maintained by other 2-pizza teams

21 2-pizza team responsibility Venn diagram THEIR PRODUCT * Deployment tools CI/CD tools Monitoring tools Metrics tool Logging tools APM tools Infrastructure provisioning tools Security tools Database management tools Testing tools. Responsible for Not responsible for *Unless their product belongs in the blue

22 We built tools to automate our software release process

23 Deployment service No downtime deployments Health checking Versioned artifacts and rollbacks

24 Pipelines Automated actions and transitions from check-in to production Development benefits: Faster Safer Consistent & standardized Visualization of the process

25 Every year we survey our software developers and in 2014 results found only one development tool/service could be correlated statistically with happier developers: Our pipelines service!

26 continuous delivery == happier developers!

27 AWS Code* services AWS CodePipeline AWS CodeStar AWS CodeDeploy AWS CodeCommit AWS CodeBuild

28 AWS Code* services Software Release Steps: Commit Build Test Production

29 AWS Code* services Software Release Steps: Commit Build Test Production AWS CodeCommit

30 AWS Code* services Software Release Steps: Commit Build Test Production AWS CodeBuild Third-Party Tooling

31 AWS Code* services Software Release Steps: Commit Build Test Production Third-Party Tooling Third-Party Tooling

32 AWS Code* services Software Release Steps: Commit Build Test Production AWS CodeDeploy

33 AWS Code* services Software Release Steps: Commit Build Test Production AWS CodeDeploy Amazon EC2 On-Premises

34 AWS Code* services Software release steps: AWS CodePipeline Commit Build Test Production

35 AWS Code* services Software release steps: AWS CodePipeline Commit Build Test Production AWS CodeCommit AWS CodeBuild Third Party Tooling AWS CodeDeploy

36 AWS CodePipeline Continuous delivery service for fast and reliable application updates Model and visualize your software release process Builds, tests, and deploys your code every time there is a code change Integrates with third-party tools and AWS

37 CodePipeline MyApplication Source Source GitHub Build JenkinsOnEC2 Jenkins Deploy JavaApp Elastic Beanstalk Action Stage Transition Pipeline

38 CodePipeline MyApplication Source Source GitHub Build JenkinsOnEC2 Jenkins NotifyDevelopers Lambda Parallel actions Deploy JavaApp Elastic Beanstalk

39 CodePipeline MyApplication Source Source GitHub Build JenkinsOnEC2 Jenkins TestAPI Runscope NotifyDevelopers Lambda Sequential actions Deploy JavaApp Elastic Beanstalk

40 Build CodePipeline MyApplication JenkinsOnEC2 Jenkins Staging-Deploy JavaApp Elastic Beanstalk QATeamReview Manual Approval Review Manual approvals Prod-Deploy JavaApp Elastic Beanstalk

41 CodePipeline MyApplication Source 1. Get changes Source GitHub Build JenkinsOnEC2 Jenkins 3. Poll for job 4. Acknowledge job 7. Put success EC2 instance 5. Get source artifact S3 Source Artifact 6. Store build artifact S3 Deploy 8. Retrieve build artifact Build Artifact JavaApp Elastic Beanstalk 9. Deploy build artifact Java App Elastic Beanstalk Web container

42 AWS service integrations Source Invoke Logic Deploy Amazon S3 AWS CodeCommit AWS Lambda AWS CodeDeploy AWS CloudFormation AWS Elastic Beanstalk AWS OpsWorks

43 We have a strong partner list, and it s growing Source Build Test Deploy

44 Extend AWS CodePipeline using custom actions Mobile testing Update tickets Provision resources Update dashboards Send notifications Security scan

45 Build & test your application

46 Building your code Building code typically refers to languages that require compiled binaries:.net languages: C#, F#, VB.net, etc. Java and JVM languages: Java, Scala, JRuby Go ios languages: Swift, Objective-C We also refer to the process of creating Docker container images as building the image. EC2

47 No building required! Many languages don t require building. These are considered interpreted languages: PHP Ruby Python Node.js EC2 You can just deploy your code!

48 Testing your code Testing is both a science and an art form! Goals for testing your code: Want to confirm desired functionality Catch programming syntax errors Standardize code patterns and format Reduce bugs due to non-desired application usage and logic failures Make applications more secure

49 Deploying your applications

50 AWS CodeDeploy Automates code deployments to any instance Handles the complexity of updating your applications Avoid downtime during application deployment Rollback automatically if failure detected Deploy to Amazon EC2 or on-premises servers, in any language and on any operating system Integrates with third-party tools and AWS

51 appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: *.html owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh

52 appspec.yml Example version: 0.0 os: linux files: - source: / destination: /var/www/html permissions: - object: /var/www/html pattern: *.html owner: root group: root mode: 755 hooks: ApplicationStop: - location: scripts/deregister_from_elb.sh BeforeInstall: - location: scripts/install_dependencies.sh ApplicationStart: - location: scripts/start_httpd.sh ValidateService: - location: scripts/test_site.sh - location: scripts/register_with_elb.sh Send application files to one directory and configuration files to another Set specific permissions on specific directories & files Remove/add instance to ELB Install dependency packages Start Apache Confirm successful deploy More!

53 Choose deployment speed and group one at a time Dev Deployment group v2 v1 v1 v1 v1 v1 Agent Agent half at a time OR Prod Deployment group v2 v2 v2 v1 v1 v1 all at once Agent Agent Agent v2 v2 v2 v2 v2 v2 Agent Agent Agent

54 DEMO!

55 General best practices used by Amazon developers CI/CD is a MUST! Commit frequently Builds on every commit Build once in a given execution flow Deploy to a running environment for further testing

56 General best practices used by Amazon developers CI/CD is a MUST! Commit frequently Builds on every commit Build once in a given execution flow Deploy to a running environment for further testing Everything that is code (application, infrastructure, documentation) goes into a repository If it s not in a repository, it doesn t go into production environments!

57 General best practices used by Amazon developers CI/CD is a MUST! Commit frequently Builds on every commit Build once in a given execution flow Deploy to a running environment for further testing Everything that is code (application, infrastructure, documentation) goes into a repository If it s not in a repository, it doesn t go into production environments! Start with continuous delivery ( gated promotion) and build up to continuous deployment once evidence of a high-level of excellence in testing is clear

58 General best practices used by Amazon developers CI/CD is a MUST! Commit frequently Builds on every commit Build once in a given execution flow Deploy to a running environment for further testing Everything that is code (application, infrastructure, documentation) goes into a repository If it s not in a repository, it doesn t go into production environments! Start with continuous delivery ( gated promotion) and build up to continuous deployment once evidence of a high-level of excellence in testing is clear Deploy to canaries, test, deploy to an AZ, test, deploy to a Region, test

59 General best practices used by Amazon developers (cont.) Code reviews are one of the best mechanisms for good code: Does this code look clean and can someone else understand it? Is the design of it meeting the expectations of its needs? Are there better/easier ways to do this same thing?

60 General best practices used by Amazon developers (cont.) Code Reviews are one of the best mechanisms for good code: Does this code look clean and can someone else understand it? Is the design of it meeting the expectations of its needs? Are there better/easier ways to do this same thing? Style checkers Will someone else in the company be able to update/fix/maintain this code?

61 General best practices used by Amazon developers (cont.) Code Reviews are one of the best mechanisms for good code: Does this code look clean and can someone else understand it? Is the design of it meeting the expectations of its needs? Are there better/easier ways to do this same thing? Style checkers Will someone else in the company be able to update/fix/maintain this code? Auto-rollbacks can be the quickest recovery mechanism after failure Rollback first, then debug what went wrong with logs/graphs/etc.

62 General best practices used by Amazon developers (cont.) Code Reviews are one of the best mechanisms for good code: Does this code look clean and can someone else understand it? Is the design of it meeting the expectations of its needs? Are there better/easier ways to do this same thing? Style checkers Will someone else in the company be able to update/fix/maintain this code? Auto-rollbacks can be the quickest recovery mechanism after failure Rollback first, then debug what went wrong with logs/graphs/etc. Thorough dashboards What is happening now? What does normal look like over some period of time? What do I do if this graph looks wrong/an alarm has been triggered? What events can I correlate with a move in a graph?

63 Code* tips and tricks All Code* products can (and should) be provisioned and managed with AWS CloudFormation! You could literally store the CloudFormation templates that provision your Code* resources in CodeCommit and update them via CodePipeline (It s like Code* inception!) Deep integration with IAM. You can assign permissions on who can commit code, approve manual approvals, deploy to certain deployment groups, and more! Integrate with AWS Lambda to do almost anything: CodeCommit has Repository Triggers CodeDeploy has Event Notifications CodePipeline has native Lambda invoke AWS CodePipeline AWS CodeDeploy AWS CodeCommit

64 aws.amazon.com/devops

65 Pradyumna Dash Solutions Architect, UK Public Sector