Esri Best Practices: Tuning, Testing, and Monitoring. Andrew Sakowicz, Frank Pizzi,

Transcription

1 Esri Best Practices: Tuning, Testing, and Monitoring Andrew Sakowicz, Frank Pizzi,

2 Process and tools

3 Process and tools Esri tools

4 Process and tools Esri tools

5 Tools download location ArcGIS Monitor - Others owner:enterpriseimp - Show ArcGIS Desktop Content

6 Enterprise Implementation Maturity Model Level Architectural Design and Capacity Planning Performance and Scalability Testing Monitoring Trend Analysis and Quantification 0 No No No No 1 Yes No No No 2 Yes Yes No No 3 Yes Yes Yes No 4 Yes Yes Yes Yes

7 Tuning

8 Tuning methodology Profile each tier starting from the top Total Response Time (t1-t2) Browser t1 Web Server t2 Wait Time ArcGIS Server Usage Time ArcSOC Search & Retrieval Time ArcSDE/DBMS

9 Profile application Fiddler measurement approximately 5.2 seconds Application performance narrowed down to specific request and map service

10 Review historical stats of the culprit service ArcGIS Monitor

11 Profile mxd of the culprit map service Mxdperfstat

12 Oracle Trace Compare elapsed time Elapsed time slightly changed due to different test runs

13 Oracle Execution plan Inefficient spatial index

14 Virtualization watch out for overallocations 88/314=0.28 cpu/vm 92/176=0.52 cpu/vm vmotion Migrations

15 Testing

16 Performance Speed, e.g. response time (seconds)

17 Scalability The ability to increase output and maintain acceptable performance

18 Capacity The maximum level of output the system can produce, e.g. X cars/sec X maps/sec At capacity Over capacity

19 Bottleneck Resource(s) limiting the performance or capacity Not bottleneck Think of : Lanes -as CPU processor Toll -as ArcGIS Server instances Cars -as map requests bottleneck

20 Step Load and Response Time Step Load (users) Response Time (sec) time

21 Throughput (request/hr) Step Load (users) Throughput(req/hr) Response Time (sec) time

22 Resource utilization: CPU, Memory, Network Step Load (users) Throughput(req/hr) CPU Utilization (%) Network used (Mbps) Response Time (sec) Memory used (Mb) time

23 Capacity User load Throughput(req/hr) CPU Utilization (%) Network used (Mbps) Response Time (sec) Capacity (~ 85% utilization) Memory used (Mb) Content length (bytes) Time

24 Testing Objectives - Meet Service-Level Agreement (SLA) - Bottlenecks analysis - Capacity planning - Benchmarking different alternatives

25 Testing process Application GIS Services Infrastructure: Hardware and Software

26 Required skill set Configuration, Tuning, Testing Tuning Configu ration Testing

27 System Test for Web GIS Test Automation ArcGIS Services - Mapping - Feature Service - OGC - Geocoding - Image Service - Network Analyst - Geoprocessing - Tile Cache Application GIS Services Infrastructure Application Testing Discipline relevant report

28 Web test tools feature comparison Tool Cost Learning Curve OS Metrics GIS Data Generation GIS Test Automation Load Runner High High Windows/Linux No No Visual Studio Medium High Windows No No JMeter Free High Requires additional plugin No No System Test Free Low Windows/Linux Yes Yes Tech Support by Esri PS as part of consulting support

29 System Test output

30 Test Results as Input into Capacity Planning Service time and Mb/tr models as input into capacity planning ST # CPU 3600 % CPU TH 100 Users TH Re sponsetime ThinkTime ST - CPU service time (sec) #CPU number of physical CPU cores %CPU - percent CPU TH throughput (tr/sec)

31 CPU capacity 1. User load: Concurrent users or throughput 2. Operation CPU service time (model) performance 3. CPU SpecRate subscript t = target subscript b = benchmark ST = CPU service time TH = throughput %CPU = percent CPU This method is used in capacity planning tools like System Designer or CPT

32 System Designer A tool for Solution Architecture design Gathering requirements Designing Capacity: CPU, Network, Memory Reporting

33 Monitoring

34 Challenges Standard monitoring tools: - Focus on infrastructure only - IT controlled - No ArcGIS components - Low success of ArcGIS troubleshooting

35 Challenges Growing complexity of ArcGIS Enterprise and underlying infrastructure Certificates Load balancer Firewall ArcGIS Web Adaptor Portal for ArcGIS Storage with immediate consistency ArcGIS Server ArcGIS Data Store Database

36 Monitoring Enterprise GIS Challenges Multiple administrators Multiple disparate monitoring/diagnostic tools Data collected in a reactive fashion: on demand and for limited time Correlation of data with different timestamp is difficult ArcGIS administrators do not have access to all tools, data and reports Challenging to quickly identify the root cause and take appropriate measures

37 Identify typical cases Overload System: Users Services

38 Identify typical cases Unstable Infrastructure: -Network -NAS -VMWare

39 Identify typical cases Bottlenecks: configuration Maintenance workflows

40 When problems arise, what is the root cause?

41 Interdependency between load, resources, performance and configuration User load Configuration Hardware resources Performance

42 Insufficient hardware resources User load Configuration Hardware resources Increased Performance Decreased

43 Increased user load or published slow services User load Configuration Hardware resources Increased Potentially Performance Decreased

44 Under configured ArcGIS server instances User load Bottleneck Configuratio n Hardware resources Increased Potentially Performanc e Decreased

45 Best practices arcgismonitor.esri.com demo site as use case

46 End to End monitoring End to End monitoring: - ArcGIS platform - Underlying infrastructure Quickly diagnose: - Unstable infrastructure - Overloaded system - Bottlenecks Scalable and non-intrusive ArcGIS Monitor used by Esri Managed Cloud Services to Monitor 500+ GIS Servers

47 Understand architecture and typical root causes Web Adaptor Category Root cause Portal for ArcGIS Web Application Configuration Updates Infrastructure ArcGIS Server Hosting Server GeoEvent Server ArcGIS Configuration Updates Infrastructure ArcGIS Data Store Infrastructure Configuration Updates Capacity (relational + tile cache)

48 Understand Root Cause and Impact analysis: culprits and victims Culprit - the most downstream failing component Victims all upstream failing components Web Adaptor Web Adaptor Victims Victims Portal for ArcGIS Portal for ArcGIS ArcGIS Server Hosting Server GeoEvent Server Culprit ArcGIS Server Hosting Server GeoEvent Server Culprit ArcGIS Data Store (relational + tile cache) ArcGIS Data Store (relational + tile cache) Example 1 Example 2

49 Map counters to architecture components Web Web Adaptor Portal for ArcGIS ArcGIS Server Hosting Server GeoEvent Server ArcGIS Data Store (relational + tile cache)

50 Map counters to architecture components ArcGIS Web Adaptor Portal for ArcGIS ArcGIS Server Hosting Server GeoEvent Server ArcGIS Data Store (relational + tile cache)

51 Map counters to architecture components Infrastructure Web Adaptor Portal for ArcGIS ArcGIS Server Hosting Server GeoEvent Server ArcGIS Data Store (relational + tile cache)

52 Categorize Alerts arcgismonitor.esri.com demo site

53 Start with Critical alerts

54 Identify culprit components ArcGIS Server GeoEvent Portal Data Store Hosting Server Portal

55 Map victims to culprit candidates Web Health are culprit indicators Further investigate SampleWorldCities performance degradation? Victims Culprits Victims ArcGIS Server ArcGIS Server Data Store Culprits Victims Hosting Server Portal Portal

56 Investigate performance degradation Performance degradation for SampleWorldCities service around 10 AM

57 Investigate typical performance degradation factors User load Hardware Capacity Tuning Configuration

58 Find correlation between performance and user load

59 Find correlation between performance and configuration Occasional 0 free instances

60 Find correlation between performance and CPU spike Select Infrastructure Check which process

61 Find correlation between performance and processes with high CPU

62 Provide Availability SLA Availability is usually expressed as a percentage of uptime in a given time span, e.g. month, year: Availability (%)=(Total time Downtime) / Total Time *100% Availability % Downtime per year Downtime per month Downtime per week 90% ("one nine") 36.5 days 72 hours 16.8 hours 2.4 hours 95% ("one and a half nines") days 36 hours 8.4 hours 1.2 hours Downtime per day 99% ("two nines") 3.65 days 7.20 hours 1.68 hours 14.4 minutes 99.5% ("two and a half nines") 1.83 days 3.60 hours 50.4 minutes 7.2 minutes 99.9% ("three nines") 8.76 hours 43.8 minutes 10.1 minutes 1.44 minutes 99.95% ("three and a half nines") 4.38 hours minutes 5.04 minutes 43.2 seconds 99.99% ("four nines") minutes 4.38 minutes 1.01 minutes 8.64 seconds

63 Provide Availability SLA For example, ArcGIS Monitor define downtime based on Critical alerts 63

64 System Test and ArcGIS Monitor

65 Please Take Our Survey on the App Download the Esri Events app and find your event Select the session you attended Scroll down to find the feedback section Complete answers and select Submit