Performance and Scalability: Tuning, Testing, and Monitoring

Transcription

1 Performance and Scalability: Tuning, Testing, and Monitoring Andrew Sakowicz, Steve McCarthy, Frank Pizzi,

2 Agenda Process, Tools, Value Performance tuning Performance testing Monitoring

3 Process, Tools, Value

4 Type Presentation Title Here Process and Tools

5 Relationship between System Tools User Load CPU% Capacity models Performance Tests

6 System Tools framework System Tools are not just tools Tool Patterns Discipline

7 System Tools overview owner:enterpriseimp Show ArcGIS Desktop Content

8 ArcGIS Monitor Demo:

9 Testing best practices

10 Definitions

11 Performance Speed, e.g. response time (seconds)

12 Scalability The ability to increase output and maintain acceptable performance

13 Capacity The maximum level of output the system can produce, e.g. X cars/sec X maps/sec At capacity Over capacity

14 Bottleneck Resource(s) limiting the performance or capacity Not bottleneck Think of : Lanes -as CPU processor Toll -as ArcGIS Server instances Cars -as map requests bottleneck

15 Test validation

16 Step Load and Response Time Step Load (users) Response Time (sec) time

17 Throughput (request/hr) Step Load (users) Throughput(req/hr) Response Time (sec) time

18 Resource utilization: CPU, Memory, Network Step Load (users) Throughput(req/hr) CPU Utilization (%) Network used (Mbps) Response Time (sec) Memory used (Mb) time

19 Capacity User load Throughput(req/hr) CPU Utilization (%) Network used (Mbps) Response Time (sec) Capacity (~ 85% utilization) Memory used (Mb) Content length (bytes) Time

20 Required skill set Configuration, Tuning, Testing Tuning Configu ration Testing

21 Tuning

22 Tuning methodology Profile each tier starting from the top Total Response Time (t1-t2) Browser t1 Web Server t2 Wait Time ArcGIS Server Usage Time ArcSOC Search & Retrieval Time ArcSDE/DBMS

23 Profile application Fiddler measurement approximately 5.2 seconds Application performance narrowed down to specific request and map service

24 Review historical stats of the culprit service ArcGIS Server Statistics Total requests Average response time Maximum response time Timeouts Maximum running instances 30 min resolution reports

25 Review historical stats of the culprit service System Log Parser

26 Review historical stats of the culprit service System Monitor ArcGIS Server Statistics

27 Profile mxd of the culprit map service Mxdperfstat

28 Oracle Trace Compare elapsed time Elapsed time slightly changed due to different test runs

29 Oracle Execution plan Inefficient spatial index

30 Mxdperfstat - WorldSQLExpress.mxd mxdperfstat10.4.exe -mxd WorldSQLExpress.mxd -scale mxdperfstat10.4.exe -mxd WorldSQLExpress.mxd -scale Compare to FGDB

31 Testing

32 Testing Objectives - Meet Service-Level Agreement (SLA) - Bottlenecks analysis - Capacity planning - Benchmarking different alternatives

33 Testing process Application GIS Services Infrastructure: Hardware and Software

34 Required skill set Configuration, Tuning, Testing Tuning Configu ration Testing

35 System Test for Web GIS Test Automation ArcGIS Services - Mapping - Feature Service - OGC - Geocoding - Image Service - Network Analyst - Geoprocessing - Tile Cache Application GIS Services Infrastructure Application Testing Discipline relevant report

36 Web test tools feature comparison Tool Cost Learning Curve OS Metrics GIS Data Generation GIS Test Automation Load Runner High High Windows/Linux No No Visual Studio Medium High Windows No No JMeter Free High Requires additional plugin No No System Test Free Low Windows/Linux Yes Yes Tech Support by Esri PS as part of consulting support

37 Demo: Dynamic Map Service

38 System Test output

39 System Designer output

40 Advanced features Transaction based Import Har Editing Network GP

41 System Monitor Overview

42 Monitoring overview

43 Monitoring Enterprise GIS Challenges Multiple administrators Multiple disparate monitoring/diagnostic tools Data collected in a reactive fashion: on demand and for limited time Correlation of data with different timestamp is difficult ArcGIS administrators do not have access to all tools, data and reports Challenging to quickly identify the root cause and take appropriate measures

44 Motivation: Growing complexity of ArcGIS Enterprise Requires dependable infrastructure Certificates Load balancer Firewall ArcGIS Web Adaptor Portal for ArcGIS Storage with immediate consistency ArcGIS Server ArcGIS Data Store Database

45 When problems arise, what is the root cause? 45

46 ArcGIS Monitor Demo:

47 Value to Customers Maximize GIS Investments Administrators: Detect, diagnose, and resolve issues with availability, configuration, performance and usage Gather actionable, quantifiable operational metrics and usage trends over time Managers: Increase communication among GIS and IT staff and senior management Reduce administration costs Users: Improve end-user satisfaction

48 Standards for effective GIS monitoring Many excellent monitoring tools on the market Few provide GIS dashboards System Monitor can be used as reference implementation

49 Common cases ArcGIS Enterprise is often a victim of:

50 Overload: -users -services

51 Unstable Infrastructure: -Network -NAS -VMWare 51

52 Bottlenecks: -configuration -maintenance -workflows

53 ArcGIS Monitor Web

54 Web Application Validation

55 HAR

56 HTTP

57 ArcGIS Monitor ArcGIS

58 ArcGIS

59 ArcSOC Optimizer

60 GeoEvent

61 Portal

62 ArcGIS Monitor Database

63 Database

64 Database Catalog view

65 ArcGIS Monitor Cloud

66 Cloud (AWS)

67 ArcGIS Monitor Infrastructure

68 System

69 Rping

70 WebGIS Health (Portal HA)

71 WebGIS Health Extension What do you get to monitor? Know about failures before system fails - Portal for ArcGIS Primary or Standby site failure - Index health of Portal for ArcGIS - Portal for ArcGIS thinks both machines are primary/standby - Hosted ArcGIS for Server s health - Publishing Services - Datastore is valid but the standby machine is down - Datastore failed over in the past five minutes - Datastore Service is not running

72 VMware Avoid over allocation and live migration of running virtual machines during work hours

73 ArcGIS Monitor Usage (tr/hr)

74 Usage (Tr/hr)

75 ArcGIS Monitor GeoInfo

76 GeoInfo

77 ArcGIS Monitor License

78 License (lmutil)

79 Williams System Monitoring Steve McCarthy

80 Williams System Monitoring Williams is one of the premier natural gas infrastructure providers in North America

81 Williams System Monitoring Williams GIS Environment Environment Overview Citrix Desktop - Average 120 Citrix ArcGIS Desktop User Support 63 Development, QA and Production Servers (mostly virtual) - 26 Production Clustered ArcGIS Server Environment Services, 164 Map, 33 GP, 1 Geometry & 1 Search Federated ArcGIS Portal Services, 199 Map, 9 GP, 1 Search, 1 Geometry & 30 Feature - 1,800 User (100 Average User)

82 Williams System Monitoring ESRI System Monitoring Tools Used Environment Overview System Monitor 3 - ExcelReports

83 Williams System Monitoring ESRI System Monitoring Tools Used Environment Overview System Monitor 3 - Alerts

84 Williams System Monitoring ESRI System Monitoring Tools Used Environment Overview System Monitor 3 - Reports Citrix Desktop Usage /12/2015 1/31/2016 3/21/2016 5/10/2016 6/29/2016 8/18/ /7/ /26/2016 Avg ArcGIS Users Peak ArcGIS Users Avg GIS Citrix Users Peak GIS Citrix Users SN Tickets

85 Williams System Monitoring ESRI System Monitoring Tools Used System Monitor 3 System Log Parser (ArcGIS & Web)

86 Williams System Monitoring ESRI System Monitoring Tools Used System Designer

87 ArcGIS Monitor

88 Please Take Our Survey on the Esri Events App! Download the Esri Events app and find your event Select the session you attended Scroll down to find the survey Complete Answers and Select Submit