Automating ArcGIS Deployments Using Chef Cherry Lin (clin@esri.com) Bill Major (bmajor@esri.com) David Cordes (dcordes@esri.com) Slides: http://bit.ly/2uieg0p
Overview What, Why, When? Esri s Chef Cookbooks Easy Cooking Demo Best Practices Case Study: Esri
What, Why, When
What is Automation and Why The linking of disparate systems and software in such a way that they become self-acting or self-regulating. Reliable, repeatable and testable deployments and upgrades - DEV TEST PROD - Moving between Data Centers - Faster Disaster Recovery Quick configuration with disposable machines and rapid auto-scaling Automate and become more efficient at administration - Focus more on GIS tasks - Prepare for larger scale and clouds - Ready-to-use tools with minimum requirements to start.
Operate Software Operation (HA, DR) Org Administration Software Configuration Content Management Software Installation Analytics
re Automation Software Operation (HA, DR) Software Configuration Software Installation Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder
Automation Org Administration Content Management Analytics Python API for ArcGIS, WebGISDR, Admin API, ArcPy, R-bridge for ArcGIS
re Automation Software Operation (HA, DR) Software Configuration Software Installation Software Setups, Enterprise Builder, Chef, Cloud formation, Cloud Builder
ArcGIS Enterprise Software Setups = + + + ArcGIS Enterprise ArcGIS Web Adaptor Portal for ArcGIS ArcGIS Server ArcGIS Data Store
ArcGIS Enterprise Software Setups No need to learn 3-party technology. Deploy exact the way you want Other Esri software Time consuming More manual steps Manual configurations following different docs One time task For customized deployments Not recommended for highly available and scalable deployments
ArcGIS Enterprise Builder ArcGIS Web Adaptor Portal for ArcGIS ArcGIS Web Adaptor Hosting Server ArcGIS Data Store (relational + tile cache)
ArcGIS Enterprise Builder All-in-one deployment One GUI setup, 5 or 6 parameters Easy to use A lot less configurations One deployment pattern for one machine quick start Not for a system that needs quickly redeployed Not a DR deployment?
ArcGIS In the Cloud Enterprise Software Setups AWS - Esri AMIs - Esri CloudFormation Templates - Python and Powershell scripts Microsoft Azure - Esri Images - Cloud Builder Highly recommend using our Images and tools - Customization based on our Images and tools - Details: AWS and Azure sessions
ArcGIS Esri ArcGIS Enterprise Chef Cookbooks Software Setups IT standard automation framework Automate all that you can manually do Deploy exact the way you want. You have full control. Other Esri software Require small up-front investments Reliable, repeatable and testable - Documented deployments - High Availability and Disaster Recovery deployments - Move between Data Centers - Potential scale-ups - Automate upgrades
re Automation ArcGIS Enterprise Software Setups Enterprise Builder Software Setups Chef ArcGIS Cookbooks Quick start Easy to use Typical deployment pattern Simple deployment Limited control Not for repeatable deployments Reliable Repeatable Testable Documented deployment Full control
Esri s Chef Cookbooks
What is Chef Chef - One of the most popular IT automation platforms - Supported on Windows and Linux - Provide tools to take on configuration files Chef Components - Cookbook - Attribute - Chef-Server vs. Client How it works - Install ready-to-use cookbooks and define the configuration - Run chef with the configuration file
ArcGIS Cookbook Recipe Recipe - Attributes - Action Property Settings: Attributes Run-list
Define You Own Attributes
Define You Own Run List
Steps to get started with a single machine deployment with Chef Check out Esri Github Chef Cookbooks website v.3.1 just released - https://github.com/esri/arcgis-cookbook/wiki/get-started-with-arcgis-cookbook Install the Chef Client (v. 12.5.1 or later) Copy ArcGIS Cookbooks to server and unzip to c:\chef - https://github.com/esri/arcgis-cookbook/releases/tag/v3.1.0 Stage Setups and License Files Edit JSON configuration file to meet your own needs - c:\chef\roles\webgis-windows.json Run chef-solo from command prompt As Administrator - chef-solo -j C:\chef\roles\webgis-windows.json
Easy Cooking Demo
Supported ArcGIS Software ArcGIS Enterprise - Portal for ArcGIS -ArcGIS Server - Enabled Roles: Hosting, Image Server, GeoAnalytics, GeoEvent Server -ArcGIS Data Store - General, spatialtemporal -ArcGIS Web Adaptor And other Esri Software: - ArcGIS Pro - ArcGIS Insight - ArcGIS Desktop - ArcGIS License Manager
Support ArcGIS Enterprise Deployment Patterns
Best Practices
ArcGIS Distributed Enterprise DeploymentSoftware Setups Running Chef on different machines in order from a workstation machine - Different property files - Runs in order as required by ArcGIS Enterprise Chef Server - Central repository - Assign roles to every machine - Run the recipes/updates accordingly - Setting up -By yourself or -Hosted Solution
ArcGIS Upgrade to Enterprise 10.5 or 10.5.1 Software Setups Required properties -New Version # -New Setups -New License File Recommend to use Chef deploy before upgrade -Easier to get the properties right
ArcGIS Work with Enterprise Other Automation Software ToolsSetups Esri ArcGIS Chef Cookbooks is ArcGIS deployment tool ArcGIS Infrastructure part of your automation Work with GIS workflow automation tools -Python API for ArcGIS -WebGISDR tool Other software deployment automation tools Whole system infrastructure automation -High Availability and Disaster Recovery deployment -Moving to different data centers
ArcGIS Chef Deployment Enterprise Options Software Setups
ArcGIS Disconnected Enterprise Environment Software Considerations Setups Make sure you have proper Esri license files staged for use Software setups in a common location for access Setting up your own Chef Server Having Chef Client installation staged for installation
ArcGIS Tricks and Enterprise Tips Software Setups Consider creating a base image for some environment - Have some processes done, e.g, setups - Have some components baked in, e.g. base map data, security updates, etc. Chef Client version 12 Set password in an environment variable -l debug for debug Attributes value of True or False, no. e.g. configure_autostart : true ArcGIS Cookbooks support enabling hosting server role and GeoAnalytics Server role. Support security configuration for ArcGIS Server.
ArcGIS Key Points Enterprise about ArcGIS Software Chef Automation Setups Automate Your Manual Work Doesn t require programming skills Need plan ahead -Design -Configuration -Testing Repeatable -Success in production environment -Less downtime in production -Faster recovery
Case Study : Esri
ArcGIS Enterprise : New Software Daily Many software changes each day. Daily testing prevents accumulation complexity.
Test Automation : Not just automated functional testing Configuration automation Software installed and configured many times per build Many platforms Many software components Many architectures Limited resources
Hundreds of Scripts Every tester had their own configuration script - Python -.Net - Java - Perl - Batch files - Etc Wasted effort Hard to maintain
ArcGIS Cookbooks to the Rescue Solution: Use our cookbooks Tested Reliable Maintained
Challenge #1: But I already have a script. Not invented here Resistance to moving away from existing scripts. Time eventually removed this barrier from most of our test teams.
Challenge #2: But I don t have time to learn a new technology... Training In groups One-on-one Troubleshooting alias
Outcome: Successful adoption, focus on the functionality Has taken time but we spend Less time maintaining and troubleshooting scripts. More time focused on functional testing.
Multi-Machine Configuration Step 1: Prepare the image
Multi-Machine Architecture Command Machine Tomcat + Web Adaptor test-datastore.dev.geocloud.com Portal test-portal.dev.geocloud.com Hosting Server test-server.dev.geocloud.com Registered Data Store test-datastore.dev.geocloud.com
Baked an Image in Three Easy Steps Install Chef Copy Cookbook System Requirements
Multiple Machine Automation Step 2: Set the Parameters
One Deployment - Many Machines - One Set of Parameters Portal Server Data Store Web Server Parameters
Multiple Machine Automation Step 3: Create Role-Specific Runlists
Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters
Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters
Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters
Run Lists - What is Run On Each Machine Portal Server Data Store Web Server Parameters
Run Lists - Federation - Back to Portal Portal Server Data Store Web Server Parameters
Multiple Machine Automation Step 4: Create Shell Script (Batch File)
Script Writing for Multiple Machines Command Machine Scripts stored on command machine Copied to test machines at deploy time Run with ssh
Multiple Machine Automation Step 5: Run Deployment Script
Run: Multi-Machine Configuration in 30 minutes
Q&A and Slides Slides: http://bit.ly/2uieg0p