The Cloud's Cutting Edge: ArcGIS for Server Use Cases for Amazon Web Services. David Cordes David McGuire Jim Herries Sridhar Karra

Size: px

Start display at page:

Download "The Cloud's Cutting Edge: ArcGIS for Server Use Cases for Amazon Web Services. David Cordes David McGuire Jim Herries Sridhar Karra"

Quentin Owen
5 years ago
Views:

1 The Cloud's Cutting Edge: ArcGIS for Server Use Cases for Amazon Web Services David Cordes David McGuire Jim Herries Sridhar Karra

2 Atlas Maps Jim Herries

uses a collection of intelligent web maps with

3 Atlas sample application The Esri Thematic Atlas Sample App is a configurable web application that uses a collection of intelligent web maps with text, graphics, and images to talk about our world.

4 Lots of maps needed, quickly

5 Key requirements Supply dozens of maps to an online atlas application Stand up dozens of map services in two weeks Enable team of map publishers Option to iterate quickly on cartography, data, popups

6 CloudBuilder delivered value to the team instantly Data lead knows the data but not CloudBuilder Cartographers on team knew color and cartography but not the data and not CloudBuilder We all knew ArcGIS server and publishing workflows None of us wanted to manage a server in a production situation CB

7 16 apps 743 items

8 Lessons Learned Same day startup of server was a huge factor in project success Teaching the team publishing methods was easy Cache local, copy up FTP, now tile packages Hundreds of cached services for the U.S. to 72k scale 6 hours to cache the US down to 72 k Use a checklist to save hundreds of hours time Whatever data you have, you re going to want additional attributes Your corporate IT network staff are your friends (DNS changes, ports, etc) Serve dynamic until popups and carto finalized It s just another server but we don t have to worry about it

9 An ongoing resource

10 Going viral David McGuire

11 Cloud Formation Demo

12 Crisis Thursday Friday Saturday Servers Down

13 Traditional Trouble Shooting Reboot Servers? Spread Services Across Local Machines? Order New Equipment? Panic!

14 Transition to the Cloud

15 Deployment 1 hr 15 mins To Stage 30 mins To Generate An AMI 15 mins To Deploy Live Servers

16 ELB ArcGIS Servers Enterprise Geodatabase Server

17 Handle Traffic

18 What We Learned Start in the Cloud Avoid internal, static machines Design for the Cloud Architecture should be cloud focused Plan Ahead Avoid emergencies

19 Aggregated Live Feeds Frequently Changing Data How Can I Deploy and Scale?

20 Amazon Cloud Servers S3 Bucket Sync Sync Internal Processing Server

21 ELB ELB Web Application Servers ArcGIS Servers

22 Hurricane PIM

23 Hurricane PIM Over 1 Million hits in 1 day on our web map Over 7 Million dynamic map requests in 1 day

24 Next Phase Linux Linux instances are cheaper and boot faster Auto Scaling Avoid manual load management Cloud Front Take advantage of global content distribution

25 Complication Internet Amazon CloudFront Distribution Auto Scaling Elastic Load Balancing TMAppsEventsS3 S3 Bucket m1.small proxy server Availability Zone B m1.small proxy server Availability Zone C VPC Subnet 1 VPC Subnet 2 Production Security Group Internal Machine US East Region

26 Resources Esri Sample Templates - Amazon Sample Templates - Aggregated Live Feeds Make sure the Show ArcGIS Desktop Content box is checked

27 Managed Services Sridhar Karra

28 Agenda Introduction Program Overview Customer Deployments - Architecture - Lessons Questions

29 Program Overview

DEPLOYMENT SECURITY DATA MANAGEMENT HARDWARE CHANGE MANAGEMENT BANDWIDTH BACKUP CACHING Access to Enterprise GIS

30 Esri Managed Services Cloud based GIS infrastructure support HIGH AVAILABILITY ARCHIVE STORAGE SOFTWARE SYSTEM DESIGN DISASTER RECOVERY AGOL INTEGRATION INFRASTRUCTURE MONITORING SCALABILITY REPORTING PERFORMANCE TESTING REDUNDANCY DEPLOYMENT SECURITY DATA MANAGEMENT HARDWARE CHANGE MANAGEMENT BANDWIDTH BACKUP CACHING Access to Enterprise GIS Expertise Scalable Resources Rapid Deployment Reduced cost of ownership Experienced, Secure, Reliable, Scalable Extending the ArcGIS platform

31 Esri Managed Services Cloud based GIS infrastructure support Platforms: - Public Cloud: AWS, Azure, Verizon, IBM - Private Cloud: Esri, Customer Private Cloud, CGI Road Ahead FedRAMP / FISMA Moderate Security Offering ArcGIS Desktop in the cloud support

32 Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Requirements Sizing / Arch. Design Proposal / ROM Contract Staging Verification Prod. Images Production Verification Monitoring Alerts / Notifications

33 Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Sizing / Requirements Proposal / ROM Contract Arch. Design User Load Users Count; Peak (Yes / No) Application GIS Services (Dynamic, Image, GP, Feature, Cached) Security SSL / AGS Tokens / Data at Rest / Encryption Availability SLA 95%, 99% or 99.9%; Monitoring (24/7) Infrastructure Public Cloud or Private Cloud; DB/Web/App Servers

34 Esri Managed Services Cloud based GIS infrastructure support Questionnaire On Call Sizing / Requirements Proposal / ROM Contract Arch. Design Benchmark PassMark; SystemCPU (AWS, Azure, Private Cloud) owner: EnterpriseImp System Designer System Test Models; Designing; Capacity Planning GIS Services; Web Applications; Service Times System Monitor System; GIS; DB; Process; Alerts / Notifications MXDPerfStat Fix MXD issues (if any)

35 Esri Managed Services Cloud based GIS infrastructure support Staging Verification Prod. Images Deployment Create Snapshot Test MXD s (MXDPerfStat); GIS Services (System Test) Update

36 Esri Managed Services Cloud based GIS infrastructure support Production Verification Monitoring Alerts / Notifications Snapshot System s Prod. Servers Application (URL) 24/7 Support DB Process

37 Many successful deployments

38 Customer Deployments

39 Oregon Imagery Explorer Search, download, use large imagery datasets Simple, easy to use web viewer Cached and dynamic image services JS Application with Clip/Zip/Ship SLA 95%

40 Oregon Imagery Explorer (Lessons) Deployed in Amazon Web Services (AWS) 40+ GIS Services (Image, Cache, GP, etc) - (Min, Max) SOC instances vs. RAM - Paging Files / Swap space Raster/Cache data: 22 TB (Approx.) - Ship drives to AWS using Import / Export - Backup before shipping drives (New!!) Server Specs: 1 x Standard Extra Large Server - 4 Cores and 15 GB RAM - EBS Volumes: 23 drives - Max EBS Volume is: 1 TB ONLY - Windows Stripe (Server OS / Software RAID) CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU Alerts/Notifications)

41 Iberdrola USA Outage Viewer JS Application Server Auto-Scaling Data Update Automation High Availability SLA 99.9% Geographic Redundancy

42 Iberdrola USA Architecture and Workflow Amazon S3 Updates Every 15 Minutes Outage Database Load Balancer Load Balancer Primary / Auto-Scaled Amazon EC2 West Cold Backup (DR Site) Amazon EC2 East

43 Iberdrola USA Outage Viewer (Lessons) Deployed in Amazon Web Services (AWS) Points to Remember when Auto-Scaling: - Plan for the Worst - Pick the right Instance Type when AS - Launch script - Contractual Cap: Max limit for Auto-Scaling - Time needed to launch a new fully functional instance - Termination Policy: Yes or No? - IIS Logs Usage logs? - Licensing? - Check with your Account Manager. Alerts and Notifications - System Counters - Data Update Scripts (ArcPy) - CloudWatch (Monitor Disk Usage, Custom Counters (service stop, system reboot), CPU Alerts/Notifications)

44 Hurricane Sandy (Usage Statistics New York)

45 Hurricane Sandy (Usage Statistics New York)

46 Hurricane Sandy (Usage Statistics Maine)

47 Hurricane Sandy (Usage Statistics Maine)

48 Hurricane Sandy Outages (New York) Peak Sandy Hours 2/10/ :30 am

49 Heat Map Fun!!

50 Maine October 29

51 Maine October 30

52 Maine October 31

53 Maine November 1

54 Maine November 2

55 Survey Enter: Extending Services in ArcGIS for Server 10 seconds Comments really welcome FYI: Slides for this presentation are available at:

HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION

HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION Steve Bertoldi, Solutions Director, MarkLogic Agenda Cloud computing and on premise issues Comparison of traditional vs cloud architecture Review of use