Unlocking Azure with Puppet Enterprise November 29, 2016
Unlocking Azure with Puppet Enterprise November 29, 2016 v2.0
Overview Introduction to Sourced Introduction to us Infrastructure as code Evolving Azure capabilities Template driven, Puppet delivered services Multi-cloud delivery through the ages..
Who are Sourced? Adopting cloud services within an enterprise requires experience Historically Sourced Group were founded in 2009 Significant Financial Services background Specialize in Configuration Management, Automation, Cloud Computing & Data Management Achieved a number of industry firsts in these fields Offices in Australia and Canada Delivery experience in Amazon Web Services, Microsoft Azure & IBM SoftLayer Major in-flight Projects 80% data center migration to AWS for a large airline Includes an Application Delivery Framework Policy and guidance to underpin this activity Development of a strategic cloud environment for a global investment bank Engage with internal stakeholders to define a public cloud environment that is capable of housing material workloads On-going assistance on the cloud journey for large Canadian telco Full business migration of electronics medical records suite of products to AWS
Our Partnerships Strategic partnerships that align with our customer-centric approach
Us Who are these guys anyway? Keiran Sweet Senior Consultant with Sourced Group Previously Puppet lead for a large financial organisation Presented at multiple Puppet conferences and camps Background Linux & UNIX System Administration and Architecture Deployment & Integration with Cloud Providers (AWS / Azure / VMware ) Puppet user since ~2008/2009 Dog Enthusiast
Us Who are these guys anyway? Pedram Sanayei Senior Consultant with Sourced Group Strong financial services focus Background Windows & VMware System Administration Designed and architected AWS and Azure environments for large financial organisations Puppet user since ~2014 Enjoys the thrill of mid-air selfies
Infrastructure As Code
Infrastructure As Code What are some of the options today in the cloud? Native API s AWS / Azure / GCE / vsphere Write your own scripts and tools to use them Abstraction Layers Puppet / Razor / Terraform / Vagrant / Fog Leverage frameworks that simplify management Vendor Native Templating Languages AWS CFN / Azure ARM Templates Express your infrastructure in JSON / YAML
Infrastructure As Code Benefits Transparency Composition of your environment is kept in source control Greater visibility of changes and history ( git log! ) Enhanced scale out, build, test and recovery capabilities New region expansion, catastrophic simulations Build an isolated production like environment for testing first Enhanced Automation opportunities Idempotency and Self Healing CI / CD Further down the stack, unit testing, contestability
Evolving Azure Capabilities
Microsoft Azure Services overview, it s more than just compute Infrastructure Services Virtual Networks, ExpressRoute, Azure DNS, Load Balancers Storage services Blobs / Tables / Queues / Files Databases & Caching Azure SQL / DocumentDB / Azure Redis Virtual Machines and PaaS Windows / Linux Virtual Machines & Azure Websites Many many many many more.
Microsoft Azure Infrastructure as Code capabilities Azure API Abstraction Layers Azure CLI SDK - Ruby /.NET / Python / NodeJS PowerShell Module Puppet Module / Terraform / Vagrant Azure Resource Manager (ARM) Templates If you are spending significant provisioning time in the Portal You aren t doing infrastructure as code.
Anatomy of the ARM Template Azure s native templating language Declare all your Azure resources in JSON Define parameters to adjust the outcome within boundaries Define Outputs that are returned to you for consumption Why? Native Templating Language Templates get the features first No tracking other projects The console creates these templates when using the Azure Portal Store the templates like any other code Use Visual Studio Code to help with development
Template driven, Puppet delivered services
I thought this was a Puppet talk? Where does Puppet sit in all of this? Representing our Azure based environment in ARM templates There isn t an Azure service for everything we need Puppet can help here; We want to also provision instances that run our own services We don t want manual intervention to achieve this We want to ensure that security is still at the forefront We want to ensure visibility throughout the process
Deployment Workflow How do we get there? Deploy Template Provision Azure Services Provision Azure VM s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning consumable services Abstract away the Operating System It s just a commodity run time Use the Templates Outputs : { } functionality to return; Deployment Summary Service Names & URLs API Endpoints
Deployment Workflow Easy, right? Deploy Template Provision Azure Services Provision Azure VM s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete Use your CICD Tooling to initiate the deployment Focus on provisioning consumable services Abstract away the Operating System It s just a commodity run time Use the Templates Outputs : { } functionality to return; Deployment Summary Service Names & URLs API Endpoints
Bootstrapping the Puppet Agent Azure Custom Script Extensions User defined code executed on instance launch Custom Script Extensions also defined in the template In this case; Retrieve the script from a URL (ie, Blob storage) Execute the script Bootstrap the Puppet agent from the master Set additional Facts (Optional) Template parameters can be passed down to the extension if / when required Failed Puppet runs == Failed Deployment
Authorising the Puppet Agent Securely signing the CSR The Puppet CA = Security for the Puppet Service Do not sign incorrectly configured instances Policy Based Autosigning Execute code to validate the incoming CSR Automatically sign certificates that are validated to have correct: Name Only sign correctly configured instances Subscription Tags
Applying your Puppet Role to the s What becomes what? Expose a custom fact that identifies the instances role $::puppet_role Classification opportunities Do this in your code Use the Puppet node classifier to assign roles Use Hiera via hiera_include() Future state Tag Specific instances with their role Azure lacking a metadata API Coming eventually Facter feature request ready and waiting.. JIRA FACT-1383 Trusted Facts Prevents re-classification
Multi-cloud delivery through the ages..
Active/Active Why? (because management said we should..) Ensure you understand the business requirements Prepare yourself for the technical complexities Note that very few applications benefit from this configuration However you build it, you will be consuming a lowest common denominator of services Beware of silver bullets (cloud brokers) Why not just play to each of the providers strengths?
Active/Active Real world If you must.. User Traffic www.application.com CDN / DDoS Protection live.application.com 50% - 50% weighted DNS + healthcheck Microsoft Azure Amazon Web Services past.azure.application.com live.azure.application.com future.azure.application.com past.aws.application.com live.aws.application.com future.aws.application.com DNS merry-go-round DNS merry-go-round Azure Load Balancer Azure Load Balancer Azure Load Balancer Elastic Load Balancer Elastic Load Balancer Elastic Load Balancer AutoScale Disabled Ready for Destruction AutoScale Enabled Scheduled & Load Serving Traffic AutoScale Disabled Ready for Scale Up AutoScale Disabled Ready for Destruction AutoScale Enabled Scheduled & Load Serving Traffic AutoScale Disabled Ready for Scale Up Internal Traditional App Tier Traditional Data Tier
Using Puppet as heterogeneous cloud glue (2.0) It s evolution baby! hiera.yaml Deploy your using CloudFormation and ARM templates Leverage a multi-provider pipeline Puppet manage your nodes Consistently bootstrap agents cloud_provider/aws.yaml Handle provider intricacies with Puppet code Leverage Roles and Profiles to deliver the same outcome Apply the same modules to your instances across providers Supplement cloud specific values from hiera cloud_provider/azure.yaml
Any questions?
References Puppet Blog Policy Based Autosigning Policy based autosigning in Azure Azure Resource Group Templates QuickStart https://github.com/azure/azure-quickstart-templates/ Microsoft Release Pipeline Model https://msdn.microsoft.com/enus/powershell/dsc/whitepapers#the-release-pipelinemodel Image Credits Silicon Valley (HBO)
Previous Presentations Using Puppet in Automated Environments Order in a world of snowflakes Sourced Group, Puppetconf 2015 Using Puppet in Dynamic Environments The Evolving Design Patterns of Puppet Enterprise Sourced Group, Puppetconf 2014 Using Puppet with Multiple Cloud Providers Using Puppet as heterogeneous cloud glue Sourced Group, Puppetconf 2012