The Cloud's Cutting Edge: ArcGIS for Server Use Cases for Amazon Web Services David Cordes David McGuire Jim Herries Sridhar Karra
Atlas Maps Jim Herries
Atlas sample application The Esri Thematic Atlas Sample App is a configurable web application that uses a collection of intelligent web maps with text, graphics, and images to talk about our world.
Lots of maps needed, quickly
Key requirements Supply dozens of maps to an online atlas application Stand up dozens of map services in two weeks Enable team of map publishers Option to iterate quickly on cartography, data, popups
CloudBuilder delivered value to the team instantly Data lead knows the data but not CloudBuilder Cartographers on team knew color and cartography but not the data and not CloudBuilder We all knew ArcGIS server and publishing workflows None of us wanted to manage a server in a production situation CB
16 apps 743 items
Lessons Learned Same day startup of server was a huge factor in project success Teaching the team publishing methods was easy Cache local, copy up FTP, now tile packages Hundreds of cached services for the U.S. to 72k scale 6 hours to cache the US down to 72 k Use a checklist to save hundreds of hours time Whatever data you have, you re going to want additional attributes Your corporate IT network staff are your friends (DNS changes, ports, etc) Serve dynamic until popups and carto finalized It s just another server but we don t have to worry about it
An ongoing resource http://atlasmaps.esri.com/arcgis/rest/
Going viral David McGuire
Cloud Formation Demo
Crisis Thursday Friday Saturday Servers Down
Traditional Trouble Shooting Reboot Servers? Spread Services Across Local Machines? Order New Equipment? Panic!
Transition to the Cloud
Deployment 1 hr 15 mins To Stage 30 mins To Generate An AMI 15 mins To Deploy Live Servers
ELB ArcGIS Servers Enterprise Geodatabase Server
Handle Traffic
What We Learned Start in the Cloud Avoid internal, static machines Design for the Cloud Architecture should be cloud focused Plan Ahead Avoid emergencies
Aggregated Live Feeds Frequently Changing Data How Can I Deploy and Scale?
Amazon Cloud Servers S3 Bucket Sync Sync Internal Processing Server
ELB ELB Web Application Servers ArcGIS Servers
Hurricane PIM
Hurricane PIM Over 1 Million hits in 1 day on our web map Over 7 Million dynamic map requests in 1 day
Next Phase Linux Linux instances are cheaper and boot faster Auto Scaling Avoid manual load management Cloud Front Take advantage of global content distribution
Complication Internet Amazon CloudFront Distribution Auto Scaling Elastic Load Balancing TMAppsEventsS3 S3 Bucket m1.small proxy server Availability Zone B m1.small proxy server Availability Zone C VPC Subnet 1 VPC Subnet 2 Production Security Group Internal Machine US East Region
Resources Esri Sample Templates - http://esriurl.com/esriawstemplate Amazon Sample Templates - http://esriurl.com/awstemplate Aggregated Live Feeds - http://esriurl.com/livefeeds - Make sure the Show ArcGIS Desktop Content box is checked
Managed Services Sridhar Karra
Agenda Introduction Program Overview Customer Deployments - Architecture - Lessons Questions
Program Overview
Esri Managed Services Cloud based GIS infrastructure support HIGH AVAILABILITY ARCHIVE STORAGE SOFTWARE SYSTEM DESIGN DISASTER RECOVERY AGOL INTEGRATION INFRASTRUCTURE MONITORING SCALABILITY REPORTING PERFORMANCE TESTING REDUNDANCY DEPLOYMENT SECURITY DATA MANAGEMENT HARDWARE CHANGE MANAGEMENT BANDWIDTH BACKUP CACHING Access to Enterprise GIS Expertise Scalable Resources Rapid Deployment Reduced cost of ownership Experienced, Secure, Reliable, Scalable Extending the ArcGIS platform
Esri Managed Services Cloud based GIS infrastructure support Platforms: - Public Cloud: AWS, Azure, Verizon, IBM - Private Cloud: Esri, Customer Private Cloud, CGI Road Ahead FedRAMP / FISMA Moderate Security Offering ArcGIS Desktop in the cloud support
Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Requirements Sizing / Arch. Design Proposal / ROM Contract Staging Verification Prod. Images Production Verification Monitoring Alerts / Notifications
Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Sizing / Requirements Proposal / ROM Contract Arch. Design User Load Users Count; Peak (Yes / No) Application GIS Services (Dynamic, Image, GP, Feature, Cached) Security SSL / AGS Tokens / Data at Rest / Encryption Availability SLA 95%, 99% or 99.9%; Monitoring (24/7) Infrastructure Public Cloud or Private Cloud; DB/Web/App Servers
Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Sizing / Requirements Proposal / ROM Contract Arch. Design Benchmark PassMark; SystemCPU (AWS, Azure, Private Cloud) http://www.arcgis.com owner: EnterpriseImp System Designer System Test Models; Designing; Capacity Planning GIS Services; Web Applications; Service Times System Monitor System; GIS; DB; Process; Alerts / Notifications MXDPerfStat Fix MXD issues (if any)
Esri Managed Services Cloud based GIS infrastructure support Staging Verification Prod. Images Deployment Create Snapshot Test MXD s (MXDPerfStat); GIS Services (System Test) Update
Esri Managed Services Cloud based GIS infrastructure support Production Verification Monitoring Alerts / Notifications Snapshot System Emails Prod. Servers Application (URL) 24/7 Support DB Process
Many successful deployments
Customer Deployments
Oregon Imagery Explorer Search, download, use large imagery datasets Simple, easy to use web viewer Cached and dynamic image services JS Application with Clip/Zip/Ship SLA 95%
Oregon Imagery Explorer (Lessons) Deployed in Amazon Web Services (AWS) 40+ GIS Services (Image, Cache, GP, etc) - (Min, Max) SOC instances vs. RAM - Paging Files / Swap space Raster/Cache data: 22 TB (Approx.) - Ship drives to AWS using Import / Export - Backup before shipping drives (New!!) Server Specs: 1 x Standard Extra Large Server - 4 Cores and 15 GB RAM - EBS Volumes: 23 drives - Max EBS Volume is: 1 TB ONLY - Windows Stripe (Server OS / Software RAID) CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU Alerts/Notifications)
Iberdrola USA Outage Viewer JS Application Server Auto-Scaling Data Update Automation High Availability SLA 99.9% Geographic Redundancy
Iberdrola USA Architecture and Workflow Amazon S3 Updates Every 15 Minutes Outage Database Load Balancer Load Balancer Primary / Auto-Scaled Amazon EC2 West Cold Backup (DR Site) Amazon EC2 East
Iberdrola USA Outage Viewer (Lessons) Deployed in Amazon Web Services (AWS) Points to Remember when Auto-Scaling: - Plan for the Worst - Pick the right Instance Type when AS - Launch script - Contractual Cap: Max limit for Auto-Scaling - Time needed to launch a new fully functional instance - Termination Policy: Yes or No? - IIS Logs Usage logs? - Licensing? - Check with your Account Manager. Alerts and Notifications - System Counters - Data Update Scripts (ArcPy) - CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU Alerts/Notifications)
Hurricane Sandy (Usage Statistics New York)
Hurricane Sandy (Usage Statistics New York)
Hurricane Sandy (Usage Statistics Maine)
Hurricane Sandy (Usage Statistics Maine)
Hurricane Sandy Outages (New York) Peak Sandy Hours 2/10/2014-11:30 am
Heat Map Fun!!
Maine October 29
Maine October 30
Maine October 31
Maine November 1
Maine November 2
Survey http://www.esri.com/events/devsummit/session-rater Enter: Extending Services in ArcGIS for Server 10 seconds Comments really welcome FYI: Slides for this presentation are available at: http://1drv.ms/1fnkptk