Introductions Who are we? Enterprise Implementation Who are you? o? New to ArcGIS Server? Currently deploying 9.3? Currently deploying IMS? 2

Transcription

1 ArcGIS Server Performance and Scalability - Testing Methodologies Andrew Sakowicz Frank Pizzi

2 Introductions Who are we? Enterprise Implementation Who are you? o? New to ArcGIS Server? Currently deploying 9.3? Currently deploying IMS? 2

3 Audience Testers Developers Project Manager 3

4 Level Beginner g / Intermediate 4

5 Schedule and Overview Performance and Scalability Testing methodology Web applications (step by step) ArcGIS Desktop Applications Exploratory tests Tips and Tricks Q Question ti and d answer period. i d Please complete the session survey! 5

6 Presentation Objectives Teaching g Step p by y Step p testing g (a ( simple p web Flex API)) Recording and creating transactions scripts Creating load test Executing Analyzing Results Managing testing efforts Test Plan, skills, level of effort Best B t practices ti Common pitfalls 6

7 Demo Example of ArcGIS Server Flex application used for parcel management by local governmentt Flex application with dynamic y data. Application provides the search and query functions as well as Geoprocessing Map Zoom/Pan Query Q task: t k P Parcell ID Query task: School GP task: Buffer Park S Search h task: k Hospital H i l 7

8 Demo Define user workflow,, transactions and think time Define user workflow to be tested Define and name GIS transactions Initial page of application Zoom map 1 Zoom map 2 Pan map 1 Pan map 2 Query tax lot with zoom to tax lot Query school with zoom to school Query hospital with zoom to Geoprocessing Define user think time between transactions 8

9 Demo Testing tasks Record user workflow based on application user requirements Create single user web test Define transactions Set think time and pacing based on application user requirements Parameterize transaction inputs Verify test script with single user Create load test Define user load Create machine counters to gather raw data for analysis Explain variable load test variables Execute 9

10 Demo 10

11 What is measured during a typical web load test Performance measured at web server tier, not a web browser not a true user experience i Browser Total Response Time (t1-t2) t2 t1 Web Server Wait Time Usage Time Search & Retrieval Time SOM SOC SDE/DBMS 11

12 How to measure true web user performance experience? Measure at web browser, e.g. stop watch or programmatically using API events Browser Total Response Time (t1-t2) t2 t1 Web Server Wait Time Usage Time Search & Retrieval Time A test is executed at the web browser. It measures web browser call s elapsed time (roundtrip between browser and data source) SOM SOC SDE/DBMS 12

13 How to measure true web user performance experience? Measure at web browser, e.g. stop watch or programmatically using API events 13

14 How to measure true web user performance experience? Measuring map rendering at web browser using API events private function oncreationcomplete():void { logger.text += gettimer() + ' ms: Application creation completed.\n'; mythematiclayer.addeventlistener(event.complete, ontilesloaded) mymap.addeventlistener(panevent.pan_start, onpanstart); mymap.addeventlistener(zoomevent.zoom_start, onzoomstart); } private function ontilesloaded(event:event):void { if (panstartflag){ TimeTileComplete = gettimer() - TimeTileStart ; TimeTileStart = 0 logger.text += "Tiles took " + TimeTileComplete + " ms to pan.\n"; panstartflag = false; } if ( (zoomstartflag){ S l ){ TimeTileComplete = gettimer() - TimeTileStart ; TimeTileStart = 0 logger.text += "Tiles took " + TimeTileComplete + " ms to Zoom.\n"; zoomstartflag= false; } } 14

15 How to measure true web user performance experience? Measuring clientclient-side graphics a att web browser using API events private function dothematicquery():void p Q y() { mygraphicslayer.clear(); querytask.url = textqueryurl.text.tostring(); logger.text += gettimer() + " ms: Starting query...\n"; TimeQueryStart = gettimer(); querytask.execute( query, new AsyncResponder( onresult, onfault )); mygraphicslayer.addeventlistener(flexevent.update_complete,graphicslayerupdated); function onresult( featureset : FeatureSet, token : Object = null ) : void { TimeQueryDone = gettimer(); var TimeQueryDiff:uint = gettimer() - TimeQueryStart; logger.text += gettimer() + " ms: Query took " + TimeQueryDiff + " ms"; logger.text += " [" + mygraphicslayer.numgraphics + " features].\n"; } 15

16 Resources The ESRI Flex and JavaScript APIs are straight forward to test Flex and JavaScript APIs utilize the REST API to access GIS data Understanding the REST API Interface will provide insight in how to approach testing Map Service Geocode Service GP Service Geometry Service Image Service Good resource to practice test scripting

17 Managing testing efforts 17

18 Performance Testing Objectives Benchmarking g to demonstrate performance p improvements Part of formal acceptance testing Capacity planning (server sizing) Validating system architecture design 18

19 Performance Testing Benefits Less costly y to fix the p problems at the earlier development stage Proactive resolution of issues before they impact on end d users Identify bottlenecks and/or underutilized resources for redeployment 19

20 Performance Testing Methodology Ensure application pp functional testing g is complete p Tune your system prior to testing Develop test plan Create test scripts Analyze results 20

21 Performance Testing Methodology Tuning Analyze performance of each request 21

22 Performance Testing Methodology Tuning example Optimize mxd with new ArcMap Analyze Tool 22

23 Performance Testing Methodology Tuning example Optimize p mxd with mxdperfstat p Item At Scale Layer Name Refresh Time (sec) Recommendations TaxlotDenseLabel 1,000 Simplify labeling, symbology: GraphicsPhase=1.42; simplify geometry and/or set label scale; convert polygon to polyline: vertices fetched=200001; simplify geometry and/or set label scale: vertices 1 93 fetched=200001; ,000 TaxlotDenseNoLabel simplify geometry: vertices 0.53 fetched=200001; Features Vertices Labeling Geography Graphics Cursor DBMS Phase (sec) Phase (sec) Phase (sec) CPU DBMS LIO ,001 TRUE ,001 FALSE

24 Performance Testing Methodology Tuning example Optimize ArcMap document and configure correct number of ArcGIS Server instances Throughpu ut (Transactiions/Hr) Corrected (Unloaded TT: 0.34 sec) (2.1 Instances/core) Uncorrected Number of Instances (Unloaded TT: sec) (1.6 Instances/core)

25 When? Start p performance testing g early y in the development p cycle Unit test UC2008 International Distributor Track 25

26 Test Plan Understand the effort required q to develop p and ensure testing is valid: Domain experts Hardware Time Skills Document user workflows Document configuration 26

27 Selecting Web Load Test Tool Test Tools Open Pros Source Cons LoadRunner No Industry Leader Automatic negative correlations identified with ith service i llevell agreements t Http Web Testing Click and Script Very good tools for testing SOA Test results stored in database Thick Client Testing Can Can be used for bottleneck analysis High Cost Test Development in in C programming l language Test metrics difficult to manage and correlate Poor user community with few available examples No Good solution for testing Citrix Wizard driven interface guides the user Can be used for bottleneck analysis Moderate to High Cost Test metrics are poor Test Development uses proprietary language Test metrics difficult to manage and correlate Poor user community with few available examples Visual Studio Test Team No Low to moderate cost Excellent E ll t T Testt Metric M t i reporting ti Test Scripting in C# or VB.NET Unit and Web Testing available Blog support with good examples Very good for bottleneck analysis No built in support for AMF No N Thi Thick k Cli Clientt options ti Moderate user community JMeter Yes Free Tool Provides only response times Poor User community with few available examples Silk Performer 27

28 Execute Ensure Only target applications are running Application data is in the same state for every test Good configuration management is critical to getting consistent load test results 28

29 Analyze Results Expected Response Time Curve Not Tuned Tuned Concurrent Users UC2008 International Distributor Track 29

30 Analyze Results Validation Compare p and correlate key y measurements Response Time (increasing, higher than initially profiled for single user) Throughput CPU on all tiers Network on all tiers Disk on all tiers Passed tests Failed test 30

31 Analyze Results Validation Lack of errors does not validate a test Requests may succeed but return zero size image Spot check request response content size 31

32 Analyze Results Reporting and Analysis Exclude failure range, g, e.g. g failure rate > 5% % from the analysis Exclude excessive resource utilization range 32

33 Analyze Results Validation Example Unexpected curve shape: Response time should be increasing. Likel root cause: Likely ca se failed or 0 size image requests. 33

34 Analyze Results Expected results example Max CPU utilization Response Time increase correlates with: User load CPU utilization 34

35 Analyze Results Failing test case Symptom: System memory is decreasing Root cause: System memory bottleneck at Web Server process 35

36 Analyze Results Failing test case Unexpected bottleneck in cache application : 100% CPU utilization. How to determined root cause? 36

37 Analyze Results Determining g system y capacity p y Maximum number of concurrent users corresponding p g to: Maximum acceptable response time First failure or 5% resource utilization greater than 85%, for example CPU Different projects define acceptance criteria (performance level of service), e.g. 95% of requests under 3 sec Max request under 10 sec 4/1/2009

38 Common pitfalls Application pp stack is not properly p p y configured g and tuned Single error in a test can lead to a cascade of failing requests Test client machine is bottleneck CPU Memory Networks (cache test in particular) 38

39 Common pitfalls configuration problem test case High CPU utilization (187%) of Web server due to authentication process (lsass.exe) UC2008 International Distributor Track 39

40 Demo Analyzing Test Case Results 40

41 ESRI Flex, JavaScript, and REST Resources Flex API JavaScript API elp/jssamples_start.htm REST API

42 References Fiddler Visual Studio 2008 Professional Edition (90(90-day Trial) ED72--4A79 4A DB0192B&displaylang=en MSDN Chapter 17 Load Load--Testing Web Applications Creating a Web Test BLOGS Bill Barnett's blog test--api api--enhancements enhancements--inin-vsts vsts sp1 sp1--beta.aspx Ed Glas's blog on VSTS load testing Sean Lumley's Blog request--propertiesproperties-with with--aweb--test web test--plugin.aspx 42

43 Performance Testing ArcMap and ArcEngine Applications Several tools can be used to automate thick client workflows HP HP LoadRunner Borland s Silk Perfomer NRG s NRG s AppLoader

44 Capturing Transactions in ArcMap Using Events Better choices are text indicators that eventually have a definitive ending (such as text stating complete, or no text at all)

45 Capturing Transactions in ArcMap Using Events Measuring pan and zoom transactions WaitBitmapAppear: lower left status text continuously update when drawing occurs, and will become blank when ended Testing software waits until the bitmap is matched before proceeding. Specify a search area when capturing the bitmap. Keep a search area small Animated icons or other moving status indicators are particularly difficult to implement (typical example is the spinning globe in ArcMap) Better choices are text indicators that eventually have a definitive ending (such as text stating complete complete, or no text at all)

46 Capturing Transactions in ArcMap Using Events Measuring pan and zoom transactions (Apploader tool example)

47 Capturing Transactions in ArcMap Using Events Measure opening ArcMap document (Apploader tool example) Wait for Window

48 Capturing Transactions in ArcMap Using Events Open ArcMap ArcMap,, Zoom, Close ArcMap Comment : Open ArcMap Launching ArcMap via specified MXD Waiting for the ArcMap window to appear Waiting for captured bitmap to appear. This bitmap is a blank status area indicating all drawing has completed. 5. Comment: Zoom 6. Move the mouse to the map scale field 7. Click the map scale field 8. Type the scale we wish to zoom to 9. Hit enter, initiating the map zoom 10. Comment: Start Zoom Transaction 11. Changing the Event Interval to 50ms for better transaction capturing 12. Begin the transaction 13 Wait 13. W it for f bitmap bit to t appear. This Thi bitmap bit is i the th grey status area at the lower left of the screen. It will become blank when the map has finished drawing 14. End the transaction 15. Changing the Event Interval back to its default 1000ms Comment: Close ArcMap 17. Closing ArcMap via shortcut 48

49 Exploratory tests 49

50 Using synthetic data for exploratory tests Benefits Controlled density Easy to script 50

51 Generating synthetic data RandomEdits -a custom gp tool available for download 51

52 Using synthetic data for exploratory tests Helps analyzing generic performance factors, e.g. Fetch time vs. # features 52

53 Summary Today y we covered: Performance and Scalability Testing methodology Web applications (step by step) ArcGIS Desktop Applications Exploratory tests Tips and Tricks Still have questions? 53

54 Additional Resources Questions, answers and information Tech Talk Outside this room right now! Other sessions ArcGIS A GIS Server S Performance P f and d Scalability Performance Factors ESRI Resource Centers PPTs, code and video resources.esri.com Social Networking ESRIDevSummit tinyurl.com/ ESRIDevSummitFB 54

55 Questions? Frank Pizzi, Andrew Sakowicz, ESRI Professional Services requests: John Graham, Chad Helm, 55

56 Demo Related and Supplemental Slides 56

57 Using Visual Studio for web testing Creating g web test script p 1. VS Recorder might not detect all parameters for ADF must validate with Fiddler or other tools may have to edit registry entries Fiddler and export to VS Web test 2. Deselect appropriate plugplug-in 3. Automated solutions (especially useful for ADF testing), e.g. Fiddler custom plugplug-in Declarative vs. Code web test Looping and if logic can be implemented only in coded web test 57

58 Scripting Stateless vs. Stateful applications Examples p Stateless:JavaScript Stateless:JavaScript,, Flex, SilverLight API Stateful Stateful::.NET and Java ADF State St t managementt Stateful Stores sessions in a web server process (InProc (InProc)) Often web server memory bound Stateless Manages g state in a browser Challenges of testing Stateful application more difficult to test 58

59 Scripting applications - Challenges Stateful application pp Test scripts must manage the state Most requests contain session based or dynamic GUID Map Images Scale bar North Arrow Search task Parameterize request 59

60 Creating Transaction Scripts manual approach Most load testing g tools include a an HTTP debugging gg g proxy to model user workflows. The output from the HTTP Debugging proxy is raw text or another format that represents the Web Client requests requests. When recording user workflows, use transaction markers between the transactions. For Example, initial page load, zoom, and pan are transactions The transaction marker is also the correct place to insert think time as a user would naturally pause after a transaction 11/16/2007

61 Creating Transaction Scripts - manual approach 11/16/2007

62 Creating Transaction Scripts manual approach Using g the transaction markers we will define GIS transactions. The content returned from the initial page request will contain t i dynamic d i images i that th t mustt be b captured, t d formatted properly, and sent back to the server in the form of an HTTP GET. All references to HTTP GET with ESRI.ArcGIS.Web.MimeImage.ashx must be altered to reflect the new session Transaction markers must be removed 11/16/2007

63 Creating Transaction Scripts Fiddler manual Using Fiddler, an HTTP Debugging proxy, record the desired Web client workflow Be sure to include the transaction markers in y your workflow. When you have completed recording the defined workflow, select all (Ctrl (Ctrl--A) and go to FileFile->Save >Save-->Session(s) >Session(s)-->as Visual Studio Web Test and save the raw HTTP traffic to fiddler.webtest. You will be prompted to select VS2008 plugins on export. Please select the following options options-- 11/16/2007

64 Creating Transaction Scripts VS Web Test Declarative webtest 11/16/2007

65 Creating Transaction Scripts VS Web Test manual Group p requests q into transactions Define extraction rules for each web request that returns dynamic images images. Extraction rules define search patterns used to parse web requests, and the results of the search are held in a variable defined in extraction rule. rule 11/16/2007

66 Creating Transaction Scripts VS Web Test manual extraction Rules 11/16/2007

67 Creating Transaction Scripts VS Web Test manual extraction rules for ADF testing Parameters Value Notes Context Parameter Name t0-i0 This item represents a variable that will be set when the extraction rule is executed on the response of an HTTP request. I recommend following the convention to the left left. T0 represents transaction0 and i0 represent image0. If there is more than one request coming back in the response of the server or the define transaction has multiple dynamic i images th the C Context t t Parameter P t name will simply be advanced. For Example, t0-i2. Regular Expression ImgID=((([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fAF]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}) ([09 fa F]){32}) 9a-fA-F]){32}) Dynamic images will have this signature, and the regular expression will find multiple images if they are present in the HTTP response from the server Ignore Case False Required True Index 0 The regular expression will return an array of item that matches the search criteria This number represents the criteria. index of the array that must be dereferenced. 11/16/2007

68 Creating Transaction Scripts VS Web Test manual extraction rules for ADF testing Now Now,, we have defined all of the extraction rules for th first the fi t transaction. t ti Next, we must find the static requests from the initial HTTP Debugging proxy and modify them to be d dynamic. i Below is an example of a static request for a mime image that must be modified. We will delete all off the query string parameters, but we must leave the headers alone. 11/16/2007

69 Creating Transaction Scripts VS Web Test manual extraction rules for ADF testing g Now,, we will use the variable defined by y the Context Parameter Name in the extraction rules As you can see, we added the {{t0{{t0-i0}} to the correct requestt parameter. t Thi modification This difi ti to t the th requestt will ill dereference the variable that has been set when the extraction rule is executed. M d li Web Modeling W b Clients Cli t 11/16/2007

70 Creating load test Visual Studio 4/1/2009

71 Creating load test Visual Studio 4/1/2009

72 Creating load test Visual Studio 4/1/2009

73 Creating load test Visual Studio 4/1/2009

74 Running a Load Test with System Monitoring While the load test is running, g, various system y components p should be monitored including the CPUs, disk subsystem, memory, and network. System monitoring applies to all the systems that play a part in the test configuration, including the database server. At a minimum, the following core components should be monitored during the tests: CPU Utilization Memory Disk Utilization Network Utilization 74

75 Creating load test Visual Studio 4/1/2009

76 Creating load test Visual Studio 4/1/2009

77 Executing load test Visual Studio 4/1/2009

78 Glossary Transactions - A transaction is defined as a Web page request that results in a Web browser output display Response R Ti Time - Response time is the accumulation of time associated with processing requests (service time) and waiting for service (queue time). Queue Time - Queue time is the amount of time spent waiting to receive service Service Time - Service time is the time required by a device (or server) to service a request Throughput - Throughput is the rate at which some amount of work is being performed Utilization - Utilization measures the fraction of time a device is busy servicing a request and is usually reported as a percentage 78

79 Glossary Scalability - Scalability is the ability of a computer system to adapt to an increasing load demand while providing minimal performance f degradation d d ti Performance - Performance is related to response time and is based on a user's observation of how well a system is performing a given task. Performance Testing Used to determine how fast some aspect p of a system y p performs under a particular p workload Stress Testing - Stress testing is a form of testing that is used to determine the stability of a given system or entity Load L d Testing T ti - Load L d testing t ti is i the th process off putting tti demand d d on a system or device and measuring its response 79