Unlocking Azure with Puppet Enterprise. November 29, 2016

Size: px

Start display at page:

Download "Unlocking Azure with Puppet Enterprise. November 29, 2016"

Cuthbert Smith
6 years ago
Views:

1 Unlocking Azure with Puppet Enterprise November 29, 2016

2 Unlocking Azure with Puppet Enterprise November 29, 2016 v2.0

3 Overview Introduction to Sourced Introduction to us Infrastructure as code Evolving Azure capabilities Template driven, Puppet delivered services Multi-cloud delivery through the ages..

4 Who are Sourced? Adopting cloud services within an enterprise requires experience Historically Sourced Group were founded in 2009 Significant Financial Services background Specialize in Configuration Management, Automation, Cloud Computing & Data Management Achieved a number of industry firsts in these fields Offices in Australia and Canada Delivery experience in Amazon Web Services, Microsoft Azure & IBM SoftLayer Major in-flight Projects 80% data center migration to AWS for a large airline Includes an Application Delivery Framework Policy and guidance to underpin this activity Development of a strategic cloud environment for a global investment bank Engage with internal stakeholders to define a public cloud environment that is capable of housing material workloads On-going assistance on the cloud journey for large Canadian telco Full business migration of electronics medical records suite of products to AWS

5 Our Partnerships Strategic partnerships that align with our customer-centric approach

6 Us Who are these guys anyway? Keiran Sweet Senior Consultant with Sourced Group Previously Puppet lead for a large financial organisation Presented at multiple Puppet conferences and camps Background Linux & UNIX System Administration and Architecture Deployment & Integration with Cloud Providers (AWS / Azure / VMware ) Puppet user since ~2008/2009 Dog Enthusiast

7 Us Who are these guys anyway? Pedram Sanayei Senior Consultant with Sourced Group Strong financial services focus Background Windows & VMware System Administration Designed and architected AWS and Azure environments for large financial organisations Puppet user since ~2014 Enjoys the thrill of mid-air selfies

8 Infrastructure As Code

9 Infrastructure As Code What are some of the options today in the cloud? Native API s AWS / Azure / GCE / vsphere Write your own scripts and tools to use them Abstraction Layers Puppet / Razor / Terraform / Vagrant / Fog Leverage frameworks that simplify management Vendor Native Templating Languages AWS CFN / Azure ARM Templates Express your infrastructure in JSON / YAML

10 Infrastructure As Code Benefits Transparency Composition of your environment is kept in source control Greater visibility of changes and history ( git log! ) Enhanced scale out, build, test and recovery capabilities New region expansion, catastrophic simulations Build an isolated production like environment for testing first Enhanced Automation opportunities Idempotency and Self Healing CI / CD Further down the stack, unit testing, contestability

11 Evolving Azure Capabilities

Microsoft Azure Services overview, it s more than just compute Infrastructure Services Virtual Networks, ExpressRoute, Azure DNS, Load Balancers Storage services Blobs /

12 Microsoft Azure Services overview, it s more than just compute Infrastructure Services Virtual Networks, ExpressRoute, Azure DNS, Load Balancers Storage services Blobs / Tables / Queues / Files Databases & Caching Azure SQL / DocumentDB / Azure Redis Virtual Machines and PaaS Windows / Linux Virtual Machines & Azure Websites Many many many many more.

13 Microsoft Azure Infrastructure as Code capabilities Azure API Abstraction Layers Azure CLI SDK - Ruby /.NET / Python / NodeJS PowerShell Module Puppet Module / Terraform / Vagrant Azure Resource Manager (ARM) Templates If you are spending significant provisioning time in the Portal You aren t doing infrastructure as code.

Anatomy of the ARM Template Azure s native templating language Declare all your Azure resources in JSON Define parameters to adjust the outcome within boundaries Define Outputs that are returned to

14 Anatomy of the ARM Template Azure s native templating language Declare all your Azure resources in JSON Define parameters to adjust the outcome within boundaries Define Outputs that are returned to you for consumption Why? Native Templating Language Templates get the features first No tracking other projects The console creates these templates when using the Azure Portal Store the templates like any other code Use Visual Studio Code to help with development

15 Template driven, Puppet delivered services

16 I thought this was a Puppet talk? Where does Puppet sit in all of this? Representing our Azure based environment in ARM templates There isn t an Azure service for everything we need Puppet can help here; We want to also provision instances that run our own services We don t want manual intervention to achieve this We want to ensure that security is still at the forefront We want to ensure visibility throughout the process

Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning consumable

17 Deployment Workflow How do we get there? Deploy Template Provision Azure Services Provision Azure VM s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning consumable services Abstract away the Operating System It s just a commodity run time Use the Templates Outputs : { } functionality to return; Deployment Summary Service Names & URLs API Endpoints

Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning

18 Deployment Workflow Easy, right? Deploy Template Provision Azure Services Provision Azure VM s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning consumable services Abstract away the Operating System It s just a commodity run time Use the Templates Outputs : { } functionality to return; Deployment Summary Service Names & URLs API Endpoints

19 Bootstrapping the Puppet Agent Azure Custom Script Extensions User defined code executed on instance launch Custom Script Extensions also defined in the template In this case; Retrieve the script from a URL (ie, Blob storage) Execute the script Bootstrap the Puppet agent from the master Set additional Facts (Optional) Template parameters can be passed down to the extension if / when required Failed Puppet runs == Failed Deployment

Execute code to validate the incoming CSR Automatically sign certificates that are

20 Authorising the Puppet Agent Securely signing the CSR The Puppet CA = Security for the Puppet Service Do not sign incorrectly configured instances Policy Based Autosigning Execute code to validate the incoming CSR Automatically sign certificates that are validated to have correct: Name Only sign correctly configured instances Subscription Tags

21 Applying your Puppet Role to the s What becomes what? Expose a custom fact that identifies the instances role $::puppet_role Classification opportunities Do this in your code Use the Puppet node classifier to assign roles Use Hiera via hiera_include() Future state Tag Specific instances with their role Azure lacking a metadata API Coming eventually Facter feature request ready and waiting.. JIRA FACT-1383 Trusted Facts Prevents re-classification

22 Multi-cloud delivery through the ages..

23 Active/Active Why? (because management said we should..) Ensure you understand the business requirements Prepare yourself for the technical complexities Note that very few applications benefit from this configuration However you build it, you will be consuming a lowest common denominator of services Beware of silver bullets (cloud brokers) Why not just play to each of the providers strengths?

Active/Active Real world If you must.. User Traffic www.application.

24 Active/Active Real world If you must.. User Traffic CDN / DDoS Protection live.application.com 50% - 50% weighted DNS + healthcheck Microsoft Azure Amazon Web Services past.azure.application.com live.azure.application.com future.azure.application.com past.aws.application.com live.aws.application.com future.aws.application.com DNS merry-go-round DNS merry-go-round Azure Load Balancer Azure Load Balancer Azure Load Balancer Elastic Load Balancer Elastic Load Balancer Elastic Load Balancer AutoScale Disabled Ready for Destruction AutoScale Enabled Scheduled & Load Serving Traffic AutoScale Disabled Ready for Scale Up AutoScale Disabled Ready for Destruction AutoScale Enabled Scheduled & Load Serving Traffic AutoScale Disabled Ready for Scale Up Internal Traditional App Tier Traditional Data Tier

Consistently bootstrap agents cloud_provider/aws.

25 Using Puppet as heterogeneous cloud glue (2.0) It s evolution baby! hiera.yaml Deploy your using CloudFormation and ARM templates Leverage a multi-provider pipeline Puppet manage your nodes Consistently bootstrap agents cloud_provider/aws.yaml Handle provider intricacies with Puppet code Leverage Roles and Profiles to deliver the same outcome Apply the same modules to your instances across providers Supplement cloud specific values from hiera cloud_provider/azure.yaml

26 Any questions?

References Puppet Blog Policy Based Autosigning Policy based autosigning in Azure Azure Resource Group Templates QuickStart https://github.

27 References Puppet Blog Policy Based Autosigning Policy based autosigning in Azure Azure Resource Group Templates QuickStart Microsoft Release Pipeline Model Image Credits Silicon Valley (HBO)

28 Previous Presentations Using Puppet in Automated Environments Order in a world of snowflakes Sourced Group, Puppetconf 2015 Using Puppet in Dynamic Environments The Evolving Design Patterns of Puppet Enterprise Sourced Group, Puppetconf 2014 Using Puppet with Multiple Cloud Providers Using Puppet as heterogeneous cloud glue Sourced Group, Puppetconf 2012

Developing Microsoft Azure Solutions

1 Developing Microsoft Azure Solutions Course Prerequisites A general understanding of ASP.NET and C# concepts Upon Completion of this Course, you will accomplish following: Compare the services available