AT&T Developer Program

Transcription

1 AT&T Developer Program Application Resource Optimizer (ARO) Testing Guide Publication Date: April 27, 2015

2 Legal Disclaimer This document and the information contained herein (collectively, the "Information") is provided to you (both the individual receiving this document and any legal entity on behalf of which such individual is acting) ("You" and "Your") by AT&T, on behalf of itself and its affiliates ("AT&T") for informational purposes only. AT&T is providing the Information to You because AT&T believes the Information may be useful to You. The Information is provided to You solely on the basis that You will be responsible for making Your own assessments of the Information and are advised to verify all representations, statements and information before using or relying upon any of the Information. Although AT&T has exercised reasonable care in providing the Information to You, AT&T does not warrant the accuracy of the Information and is not responsible for any damages arising from Your use of or reliance upon the Information. You further understand and agree that AT&T in no way represents, and You in no way rely on a belief, that AT&T is providing the Information in accordance with any standard or service (routine, customary or otherwise) related to the consulting, services, hardware or software industries. AT&T DOES NOT WARRANT THAT THE INFORMATION IS ERROR-FREE. AT&T IS PROVIDING THE INFORMATION TO YOU "AS IS" AND "WITH ALL FAULTS." AT&T DOES NOT WARRANT, BY VIRTUE OF THIS DOCUMENT, OR BY ANY COURSE OF PERFORMANCE, COURSE OF DEALING, USAGE OF TRADE OR ANY COLLATERAL DOCUMENT HEREUNDER OR OTHERWISE, AND HEREBY EXPRESSLY DISCLAIMS, ANY REPRESENTATION OR WARRANTY OF ANY KIND WITH RESPECT TO THE INFORMATION, INCLUDING, WITHOUT LIMITATION, ANY REPRESENTATION OR WARRANTY OF DESIGN, PERFORMANCE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, OR ANY REPRESENTATION OR WARRANTY THAT THE INFORMATION IS APPLICABLE TO OR INTEROPERABLE WITH ANY SYSTEM, DATA, HARDWARE OR SOFTWARE OF ANY KIND. AT&T DISCLAIMS AND IN NO EVENT SHALL BE LIABLE FOR ANY LOSSES OR DAMAGES OF ANY KIND, WHETHER DIRECT, INDIRECT, INCIDENTAL, CONSEQUENTIAL, PUNITIVE, SPECIAL OR EXEMPLARY, INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF BUSINESS INFORMATION, LOSS OF GOODWILL, COVER, TORTIOUS CONDUCT OR OTHER PECUNIARY LOSS, ARISING OUT OF OR IN ANY WAY RELATED TO THE PROVISION, NON-PROVISION, USE OR NON-USE OF THE INFORMATION, EVEN IF AT&T HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH LOSSES OR DAMAGES. i

3 Table of Contents 1 Introduction Overview Testing Strategy User Scenarios Testing Stages Testing Flow Testing with ARO Data Collection Killing Tasks Saving and Naming Traces Data Dump Analyzing Results Capturing Application Traces Using ARO Trace Collection Basics Trace Execution Timing Testing Network Activity Logging into Outside Applications for Authentication Killing Tasks and Starting the Data Collector Collecting Traces Based on Application Type Basics Social Media GPS and Mapping Games Third Party App Stopping the ARO Data Collector ARO Test Cases ii

4 Table of Contents 5.1 Text File Compression Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Duplicate Content Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Cache Control Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Content Expiration Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Content Pre-fetching Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Combine JS and CSS Requests Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Resize Images for Mobile Test Testing strategy and suggestions iii

5 Table of Contents Suggested Best Practices Test Results and Severity Minify CSS, JS, and HTML Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Use CSS Sprites for Images Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Connection Opening (Self Test) Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Unnecessary Connections Multiple Simultaneous Connections Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Inefficient Connections Periodic Transfers Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Inefficient Connections Screen Rotation Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity iv

6 Table of Contents 5.14 Inefficient Connections Connection Closing Problems Testing strategy and suggestions Suggested Best Practices Test Results and Severity Inefficient Connections Offloading to WiFi when Possible Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity ,500 HTTP Status Response Codes Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity ,302 HTTP Status Response Codes Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity rd Party Scripts Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Asynchronous Load of JavaScript in HTML Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity HTTP 1.0 Usage Test Testing strategy and suggestions v

7 Table of Contents Suggested Best Practices Test Results and Severity File Order Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Empty Source and Link Attributes Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity FLASH Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Display: None in CSS Test Testing strategy and suggestions Suggested Best Practices Test Results and Severity Accessing Peripheral Applications Test Testing Strategy and Suggestions Suggested Best Practices Test Results and Severity Appendix I ARO Test Suite Capturing Application Traces Using ARO Basics Social Media vi

8 Table of Contents GPS Mapping Games Third Party App ARO Test Cases Duplicate Content Test Form Cache Control Test Form Content Expiration Test Form Inefficient Connections Periodic Transfers Test Form Inefficient Connections Screen Rotation Test Form Inefficient Connections Connection Closing Problems Test Form Accessing Peripheral Applications Test Form vii

9 Table of Tables Table 6-1: Collecting Traces for a Basic Appp Table 6-2: Collecting Traces for a Social Media App Table 6-3 Collecting Traces for a GPS - Mapping App Table 6-4 Collecting Traces for a Game App Table 6-5 Collecting Traces for a Third Party App viii

10 1 Introduction The AT&T ARO Testing Guide provides information on the basics of mobile application testing and describes how ARO fits into an overall testing strategy. It outlines a testing flow that includes steps for generating traces from different types of applications, and describes, in detail, each of the AT&T ARO best practice test cases. The appendix of the guide lists each of the test cases in the format needed to include them in a test suite. Page 1 of 57

11 2 Overview The AT&T Application Resource Optimizer (ARO) is a powerful analytical tool that compliments your existing testing procedures. By analyzing trace data captured during your test passes, ARO evaluates the way that your app handles connections, data, peripheral applications, and battery resources. The analysis provided by AT&T ARO and our suggested best practice recommendations are not specific to any one network or platform. This means that ARO best practices will help your application run faster and more efficiently on all mobile platforms and wireless networks. AT&T ARO has two components: ARO Data Collector: The Data Collector captures the data traffic of a mobile device and stores that information in trace files. ARO Data Analyzer: The Data Analyzer evaluates trace data and generates statistical and analytical results based on recommended best practices. The test results in the ARO Data Analyzer do much more than just flag pass and fail, they provide extensive analytical, graphical, and statistical data to help you identify the issue more precisely, and in some cases, suggest the fixes that should be made. By measuring how efficiently your app uses network resources, AT&T ARO is perfectly suited for helping your app pass industry certifications, preload requirements, or meet specific levels of quality assurance. In addition to testing complete mobile applications, AT&T ARO can be used to test API frameworks, SDK's, or third party pieces of applications. ARO is also available as open source code so that it can integrate into automated testing suites. 2.1 Testing Strategy To get the most useful analysis of your application when using AT&T ARO, it is important to capture trace files that accurately represent the ways in which a user uses your application. To do this, you need to develop a testing strategy that you can follow when you are capturing trace data. The testing strategy should be targeted to your application, and should consider questions such as the following: What are the different types of users that will use your application, and how will they use it? At what speed do users actually execute the commands in your application? Page 2 of 57

12 2.2 User Scenarios 2.3 Testing Stages What are the features in your application that will generate the most network activity? Does your application use outside applications, and how can you test for issues that this might cause? Does your application consistently use device peripherals, and how can you test these specifically? AT&T ARO was created to assist developers with evaluating real world use of their applications. Of course, not all users will approach an application in the same way or use it to the same extent. Some users may actively use all of the features of your application and keep it open for a long time. Some may only use one specific feature, or use your application very quickly. Still others may open your application and allow it to run in the background with little user interaction. When using the ARO Data Collector it s important to capture these different scenarios in separate traces. Separating these user scenarios into distinct traces helps you narrow down any areas that you would like to improve when you analyze your application. Creating user scenarios also helps you understand how your users are approaching your product, and how your application serves the needs of its customers. As your app is being designed and developed, and even after it is released, you can use AT&T ARO in each testing stage. The earlier you can use ARO in your development, the better test early and test often. ARO is a very effective tool for comparing the iterative changes that occur during the development of an app. The following describes how ARO fits into some common testing stages: Code Development: The suggested development practices contained in the Best Practices on the AT&T ARO site give you recommendations about development choices you can make to optimize your app from the start. Unit, Integration, and System testing: As soon as you have a section of code that can run on a network, you can at least analyze the packet transfers using ARO. In this way you can test the performance of units of code, and as you integrate the units of code together, and certainly when you have an entire system running, you will be able to analyze even more data from your app. Test Lab: The ARO Test Cases in the appendix of this document are designed to be used in a test lab environment. Page 3 of 57

13 2.4 Testing Flow QA Testing: ARO gives you a quick view into how you have implemented your development best practices. Customer Acceptance and Quality Validation of pre-loads: This is the kind of optimization testing that ARO was designed for. Following a testing strategy based on the general ideas and techniques described in the previous sections, use AT&T ARO as a compliment to your existing test suite by following this basic flow. Generate traces while testing the functionality of the app using your existing test suite. Generate an active trace and an idle (or background) trace. Use the instructions in Capturing Application Traces Using ARO and follow the steps in Collecting Traces Based on Application Type to tailor the trace collection based on the basic type of your app. 1. Analyze the trace results using the ARO Data Analyzer and make fixes based on the analysis. For more information on all of the test cases that ARO runs during analysis, see ARO Test Cases. 2. Generate new traces to test the fixes, and do further analysis. 3 Testing with ARO The AT&T ARO Data Collector captures the data traffic of mobile devices. As data streams across the network during a given period, the Data Collector captures each TCP packet and matches the packet information with recorded video of what the user is seeing on the device. The Data Collector also captures the radio transmission activity of the app (when the radio is active, when it is idle, etc.) by charting the Radio Resource Control (RRC) state machine, and by capturing the usage of all peripheral applications. All of this information is stored in trace files that are used by the AT&T ARO Data Analyzer to create a comprehensive analysis of a mobile application. The following section provides information on how to install and use the AT&T ARO Data Collector, how to use the ARO Task Killer in the Data Collector to stop all other tasks on the device so that you are only gathering trace data from the application being tested, and how to save and name the traces. The section also includes information about the AT&T ARO Data Analyzer, in particular, how to export all of your collected trace data to a.csv file for your own statistical analysis, and where to find detailed information on how to interpret the trace results. Page 4 of 57

14 3.1 Data Collection There are several methods for collecting trace data that can be analyzed using the AT&T ARO Data Analyzer, a summary of the most common methods are listed in this section. For a list of prerequisites and basic instructions on how to collect traces using these methods, see Collecting Data at the following location: For more detailed instructions, see the AT&T ARO User Guide at the following location: The following methods can be used to capture trace files for analysis using the AT&T ARO Data Analyzer: Android Emulator: AT&T ARO interfaces directly with the Android Emulator. By opening the Data Collector menu in the AT&T ARO Data Analyzer, you can connect to the Android Emulator and test an Android application in an emulated environment on your computer. ARO Data Collector APK: The AT&T ARO Data Collector APK or the ARO Open Source APK can be installed directly on a test device for which you have administrator access. ARO Non-Rooted Data Collector - VPN Solution: By connecting a Samsung Galaxy S5, Google Nexus 5, or Google Nexus 6 to a computer running AT&T ARO, and selecting the Start Collector option, an ARO VPN is installed on the device which allows for the capture of HTTP and UDP packet data requests before they are sent, and responses when they are received. In addition to TCP and UDP packet data, the ARO VPN captures a video of the trace, and device state information, such as battery, WiFi, and GPS usage. ARO Non-Rooted Data Collector - Virtual WiFi Solution: By connecting an Android device via USB to a computer with ARO installed, and by ensuring that both are connected to the same virtual WiFi network, you can capture packet data and a video of the trace from the device to the computer and analyze the trace using the ARO Data Analyzer. ARO Windows 8 Data Collector: The ARO Windows 8 Data Collector can be downloaded from the Microsoft Visual Studio Gallery to collect traces from your application on a 32-bit Windows 8 notebook, desktop or x86 based tablet, or on a 64-bit desktop or notebook. PCAP Packet traces: PCAP packet trace files contain basic - network only data that can be captured using several different tools. Once captured, PCAP files can be opened directly in the AT&T ARO Data Analyzer. Page 5 of 57

15 3.2 Killing Tasks These methods of data collection can be used to test applications on the various device platforms as follows: To use the AT&T ARO Data Collector to test applications on the Android platform: Install the AT&T ARO Data Collector APK or the ARO Open Source APK on a test device and capture data directly. Use the AT&T ARO Data Collector to access the Android Emulator and test in an emulated environment. Use the AT&T ARO Data Collector VPN Solution to capture packet data and video without requiring root access to the test device. Use the Virtual WiFi Solution to capture packet data and a video of the trace. To use the AT&T ARO Data Collector to test applications on the Windows 8 platform: Download and use the ARO Windows 8 Collector. To use the AT&T ARO Data Collector to test applications on other platforms: Use an Android test device with the AT&T ARO Data Collector APK installed, as a wireless access point to collect a trace from a non-android device. Collect a PCAP Packet trace. For more information on collecting traces on non-android platforms, including how to sync your own video to a PCAP Packet trace, AT&T ARO User Guide at the following location: To use the ARO Task Killer to stop all tasks except the one you are testing, do the following: 1. Launch the ARO Data Collector on the Android device. 2. Select "Open Task Killer." 3. Select all tasks. 4. Select "Kill Selected Tasks." These steps will stop all tasks except any exempted System processes. Note: ARO requires Super User access in order to run. If the application you are testing also requires Super User access, then ignore the task killer step and eliminate specific IPs during analysis. Page 6 of 57

16 3.3 Saving and Naming Traces 3.4 Data Dump The trace folder name must not contain special characters or spaces. For instance, the folder name MyTest001 is valid, but the folder name My Test 001 or My!Test is not valid. Limit the folder name to alphanumeric characters. The ARO Data Analyzer has a Data Dump feature that makes it easy for you to combine all of the data from several trace files into a.csv file. When you open the Tools menu in the Data Analyzer and select the Data Dump option, a dialog is opened that allows you to select any number of trace folders (folders that contain trace files). When you click Open, the dialog will prompt you for a file name and a location for a.csv file that will contain a data dump of all the information in each of the selected trace files, combined into one set of data. 3.5 Analyzing Results After executing the testing strategy that you have designed for your application, refer to the AT&T ARO Analysis Guide at for information on how to interpret the results of your tests in the AT&T ARO Data Analyzer. 4 Capturing Application Traces Using ARO The following section contains specific test instructions for collecting application traces using ARO. The test instructions differ based on the basic category of the application being tested, because of the differences in functionality, use of peripheral applications, and how a user would typically use that application type. It s understood that these application categories are broad and that many applications combine elements from more than one category, but these testing instructions are meant to offer an approach for testing common sets of functionality. To use these instructions, identify which category (or categories) the application belongs to, and follow the steps listed for each category to generate both of the following trace types: Active trace: Collect this trace while moving actively through the functionality of the application. Idle (or Background) trace: Collect this trace after starting the application and leaving it completely idle for 30 minutes. This type of trace provides important data about improperly closed connections and peripheral usage. Page 7 of 57

17 In addition to the type of application being tested, consider the following basics when collecting your traces. 4.1 Trace Collection Basics There are several things to consider when collecting an application trace, such as, the timing in which you execute the commands in the app, the functionality in the app that you want to focus on specifically, and how third party applications will affect your trace. The following sections describe these considerations in more detail Trace Execution Timing The timing in which you execute key presses and maneuver through your application when capturing a trace is important. If you are moving through screens too quickly, you may not be able to identify differences in the downloading of your content. If you move too slowly, you will not reflect a true user experience and you may misrepresent the energy usage of your app due to radio state changes and timeouts that wouldn t typically occur Testing Network Activity When approaching AT&T ARO to begin creating traces from your application, you should review the functionality in your application for features that will create the most network traffic. Some of these may already be obvious: streaming video, uploading pictures, or viewing new content the first time a page is displayed. Some of the less obvious activity can be found easily by watching the network indicator on the top notification bar of your device Logging into Outside Applications for Authentication If your application uses another application for login authentication, be aware that it may awaken that application in the background during your trace. For example, if your application allows users to sign-in with the login from an external application, the external application or the browser may become active in the background. By testing your application with and without these other applications running, and comparing the traces, you can determine how the performance of these other applications running in the background affects the experience of your users. Page 8 of 57

18 4.2 Killing Tasks and Starting the Data Collector To begin collecting traces using the ARO Data Collector, complete the following steps. 1. Launch the ARO Data Collector on the Android device, and kill all unneeded tasks by doing the following. a. Select "Open Task Killer". b. Select all tasks in the task list. c. Select "Kill Selected Tasks." When this step is completed, it is expected that all tasks have been stopped (except exempted System processes). Please refer to the section Killing Tasks for more information. 2. On the Android device select "Start collector", then select "Hide Collector." The ARO Data Collector will now start collecting the trace. 4.3 Collecting Traces Based on Application Type Basics Now that the ARO Data Collector is collecting a trace, follow the instructions in one or more of the following sections based on the type of application you are testing. For a basic application that does not have specific functionality for social media, GPS, mapping, or games, gather an Active and Idle trace as follows. Active Trace: On the test device, actively use the App being tested for about 20 minutes. Cover all the states that are applicable to the App, including: GPS State Bluetooth State Camera State Screen Rotation Social Media Idle Trace: On the test device, launch the App being tested and leave it Idle for 30 minutes. For an application with social media functionality, gather an Active and Idle trace as follows. Page 9 of 57

19 Active Trace: On the test device, actively use the App being tested for about 20 minutes. Test all of the following areas (General, Camera, GPS, and Chat) that apply to the App being tested, using the instructions for each area as follows: General: 1. Login using a valid and password. 2. Sync Device data with the app. For example, Calendar, Gallery, Contacts, whichever is applicable for the app. 3. Update the Status or Tweet whichever is applicable. 4. After the Home page loads, do a screen rotation (i.e., from Portrait to Landscape mode and vice versa). This step is not applicable if you are collecting a trace via hotspot for a non-android device. 5. Add a Friend, Delete a friend, and Create a group if applicable. 6. Go to the friend's page or profile. 7. Exercise the Search function by finding a friend, group, etc. 8. Comment and Like a status, tweet, picture, video, and so on. 9. Go to the App Settings page and make changes (if applicable) and apply them. Camera: 1. Click on the Camera button in the app. Take a new picture or video, Upload and Share it. 2. Tag a Friend and Update the location if applicable. 3. Download a picture or video and save it on the device. Location: Chat: 1. Click on the Location update or Check-in button. Enter a location or select from the suggested list and update the location. (Make sure that GPS is set to ON in the Device Location settings). (Note: To test this area, set up another user on a different device.) 1. Select a friend. 2. Send a message, picture, video, and emoticons. 3. Using another device and account, send a message, picture, video, and emoticons to the device where ARO is running. 4. Change your presence status. For example, idle, busy, invisible, offline, etc. 5. Using another device and account, change that user's presence status. Changes should be displayed in the ARO device. Page 10 of 57

20 6. If applicable, exercise a group chat. 7. Once all of the functionalities are tested, Logout of the account. Idle Trace: On the test device, launch the App being tested, Login before starting ARO, Start ARO, and leave it Idle for 30 minutes GPS and Mapping Games For an application with GPS and mapping functionality, gather an Active and Idle trace. Remember that actual GPS does not work indoors so if you are testing it specifically you should ensure that the app is not falling back to another method of finding location. Active Trace: On the test device, actively browse the App being tested for about 20 minutes. Cover all the states that are applicable to the App, including the following GPS and Mapping features: 1. Turn on "Use GPS Satellites" and "Use Wireless networks" in the Location Services Settings. 2. Turn on WiFi. 3. Search for a location, such as "Seattle, WA". 4. Use the device zoom facility to make areas legible. 5. Select "Get Directions." 6. Select "Navigation." 7. Use App Settings to see alternate routes, list directions, view layers (Sat, Map), and so on. 8. Toggle Voice narration. 9. Select App Settings "Help." 10. If time allows, perform an actual navigation trip. 11. Select App Settings / About. 12. Select "Exit Navigation". Alternately, select the "Back" key. Idle Trace: On the test device, launch the App being tested and leave it Idle for 30 minutes. For a Game application, gather an Active and Idle trace as follows. Active Trace - On the test device, actively launch and play the Game App being tested for about 20 minutes. Cover all the states that are applicable to the Game App, including: 1. New Game, Saved Game. 2. Set parameters: i.e., Difficulty: Easy, Normal, Hard. Page 11 of 57

21 3. Settings/Options: toggle and set items. 4. Screen Rotation, if applicable. 5. Play the game to complete a level or expire, whichever comes first. 6. Exit after saving the session. 7. View the About screen and quit the Game app. Idle Trace: On the test device, launch the Game App being tested and leave it Idle for 30 minutes Third Party App If the application being tested, uses functionality from a third party App (such as, advertising), gather an Active and Idle trace on that third party functionality specifically so that you can better determine how its performance affects the application that contains it. The following steps describe a test for the advertising app called Flurry. For an active trace, to collect data on the test device, actively launch the third party App being tested for about 5 minutes. Cover all the states that are applicable to the App, including: 1. View Splash Screen. 2. Select Settings (hard key/soft key). 3. In Settings Menu, select "Settings" and then select each item in the resulting list: For example, (in Flurry): Continue Age Gender 4. Select OK or Cancel (for example), if applicable, for each item listed. 5. Exercise the basic functionality of the app. 6. Select "Back." 7. In the Settings Menu, select "List" and exercise each item in the resulting list. 8. Select "Back". 9. Exit by selecting "Back". For an idle trace, On the test device, launch the third party App being tested and leave it Idle for 30 minutes. Page 12 of 57

22 4.4 Stopping the ARO Data Collector After the traces are collected, follow these steps to stop the ARO Data Collector. 1. In the Android device, Select "Stop Collector." ARO Collector Stops collecting the Trace. 2. Connect the Android device to a PC using USB cable and transfer the appropriate Traces. Traces should be transferred to PC successfully. 3. Launch ARO Analyzer and open the Traces. Traces should open in Analyzer and display all the data collected. 5 ARO Test Cases The AT&T ARO Data Analyzer conducts a battery of tests on your trace data. Each test provides more than just a pass/fail designation, the ARO Data Analyzer reports detailed results and analysis to help you verify that your application is following the development best practices suggested by these tests. By adding specific tests to your testing strategy for best practices that impact your application the most, you can get the greatest benefit from the results in the ARO Data Analyzer. The following sections describe each of the ARO Test Cases. The tests are focused on the main areas: File Download, Connections, HTML, and Other. Each of these sections contains tests that reinforce development best practices. For some of the key tests we have listed levels of severity beyond a basic pass/fail threshold. Other tests register a warning when they fail, and are purely informative. They are meant to suggest programming best practices that can optimize the performance of an application. 5.1 Text File Compression Test This test is conducted by checking to see if any text files larger than 850 bytes that were sent by the app were uncompressed. Sending compressed files over the network will speed delivery, and unzipping files on a device is a very low overhead operation. To increase speed, an application should compress all text files that are sent over the network Testing strategy and suggestions There is no specific strategy for this test. Page 13 of 57

23 5.1.2 Suggested Best Practices See Text File Compression for suggested best practices concerning this issue Test Results and Severity The Text File Compression test has the following possible results. Pass: When the Text File Compression test passes, it means that 0 text files sent by the app that were larger than 850 bytes were uncompressed. Fail: When the Text File Compression test fails, it means that 1 or more text files sent by the app that were larger than 850 bytes were uncompressed. The failure information in the ARO Data Analyzer will display the total amount (in KB) of text files larger than 850 bytes that were sent without compression, and includes a table of the uncompressed files that were detected. When you click on any of the files, you will see the Diagnostics Chart with the time position set to the time when that file was sent. This test does not have different severity levels. 5.2 Duplicate Content Test This test measures duplicate content. Excess duplicate content means that content was downloaded multiple times, which leads to slower applications and wasted bandwidth. To limit the amount of duplicate content, an application should cache or retain frequently used downloadable content, such as images, audio, video, or XML files Testing Strategy and Suggestions Testing for duplicate content may occur naturally as part of an active user scenario, but you may want to create a trace that focuses on duplicate content in a specific area of the application. When creating a trace to focus on duplicate content, try to navigate to the same screen a few times. This might be a home screen, a newsfeed, or playlist. Create a user scenario in which you navigate to a main screen that you suspect might be responsible for duplicate content in your traces, then, navigate away from the screen and return to it again later in the trace. You may even want to vary the time between the navigation back to the screen, to see how caching is being affected Suggested Best Practices See Duplicate Content for suggested best practices concerning this issue. Page 14 of 57

24 5.2.3 Test Results and Severity The Duplicate Content test has the following possible results. Pass: When the Duplicate Content test passes in the ARO Data Analyzer, it means that less than 3 duplicate items were downloaded. Fail: When the Duplicate Content test fails, it means that three or more items were downloaded in a duplicate manner in the loaded trace files. The failure information in the ARO Data Analyzer displays the number of duplicate items and the percentage of total content that was duplicated. When you click on the failure information, you will be directed to the Duplicate Content table of the Overview tab for further details. The following severity levels apply to this test: Sev3: If there are more than 3 files downloaded in a duplicate manner, but less than 5% of content or less than 0.5 MB of data. Sev2: If Duplicate content is >5% of total content downloaded, or >0.5 MB of data. Sev1: If Duplicate content is >20% OR > 2.0MB of data. 5.3 Cache Control Test This test checks for the presence of cache headers. For all content that should be stored in the cache, the suggested best practice is to make sure that your server is adding the appropriate cache headers, as this prevents files from being downloaded in a duplicate manner Testing Strategy and Suggestions To specifically test caching in your application, you may or may not choose to clear all user and cached data prior to beginning a trace. If you have performed a trace on a device that has not launched your application before, or if you have cleared the cached data on the device, you should execute the trace again to see if there are any performance differences between the traces Suggested Best Practices See Cache Control for suggested best practices concerning this issue Test Results and Severity The Cache Control test has the following possible results. Page 15 of 57

25 Pass: When the Cache Control test passes, it means that the data in the application is populated with cache headers. Warning: When the Cache Control test fails, it means that there were not enough cache headers present in the data. The failure information in the ARO Data Analyzer will display the percentage of files without cache headers. When you click on the failure information, you will be directed to the HTTP Cache Statistics section of the Statistics tab for further details. Even though this test registers a Warning when it fails, the following severity levels apply: Sev3: If 10% of files have no cache header. Sev2: If >25% of files have no cache header. Sev1: If >65% of files have no cache header. 5.4 Content Expiration Test The Content Expiration test is conducted by checking to see if the amount of not expired duplicate data is greater than the amount of not changed data (data for which a 304 response is received) in the loaded trace files. When this occurs, it s an indication that the application may not be using the directives and expiration times in the HTTP 1.0 expiration model correctly Testing strategy and suggestions To generate an application trace that will test your expiration settings, follow the same strategy as you would to test caching Suggested Best Practices See Cache Expiration for suggested best practices concerning this issue Test Results and Severity The Content Expiration test has the following possible results. Pass: When the Content Expiration test passes, it means that It appears that content expiration is being handled properly, and that no caching issues were detected in the trace. Warning: When the Content Expiration test fails, it means that the amount of not expired duplicate data is greater than the amount of not changed data in the loaded trace files. Page 16 of 57

26 The failure information in the ARO Data Analyzer will display the number of duplicate files with not expired cache headers and the number of not changed data 304 responses. When you click on this information, you will be directed to the HTTP Cache Statistics section of the Statistics tab for further details. Even though this test registers a Warning when it fails, the following severity levels apply to this test only if Duplicate Content is detected: Sev3: If the number of duplicates is greater than the number of 304 responses. Sev2: If there are > 3 duplicates and zero 304 responses. Sev1: If there are >25 duplicates and zero 304 responses. 5.5 Content Pre-fetching Test The Content Pre-fetching test is conducted by checking to see if there is five or more user input bursts (sequential requests for packet data initiated by the user) in a row, in the loaded trace files. When this occurs, it s an indication that there may be content which should be pre-fetched to improve the perceived speed of the application. Downloading files as needed, can slow the user experience. If a user scrolls through the main screen of the application and has to wait for images to load, the application appears slow Testing Strategy and Suggestions To generate an application trace that will test pre-fetching, follow the same strategy as you would to test caching. For information about cache testing, see Section 5.2 Cache Control Test Suggested Best Practices See Content Pre-fetching for suggested best practices concerning this issue Test Results and Severity The Content Pre-fetching test has the following possible results. Pass: When the Content Pre-fetching test passes, it means that the files in the trace seem to be downloaded in reasonable bursts, although this may need to change as user behavior changes. Fail: When the Content Pre-fetching test fails, it means that there are five or more user input bursts in a row in the loaded trace files. Page 17 of 57

27 The failure information in the ARO Data Analyzer displays the number of user input bursts in a row. When you click on this information, you will be directed to the Burst chart in the Diagnostics tab for further details. This test is primarily informational and is meant to suggest a development best practice, so it does not have different severity levels. 5.6 Combine JS and CSS Requests Test The Combine JS and CSS Requests test is conducted by checking for multiple requests for CSS files within 2 seconds, or multiple requests for JavaScript files within 2 seconds. Multiple requests for separate files of the same type can slow the loading of a page. Whenever possible, files of the same type, such as JavaScript and CSS, should be combined into as few files as is practical for an app Testing strategy and suggestions Ensure that all pages in the app are loaded, and as much functionality as possible in the app is used, so that any JS and CSS requests will be exercised Suggested Best Practices See Combine JavaScript and CSS Requests for suggested best practices concerning this issue Test Results and Severity The Combine JS and CSS Requests test has the following possible results. Pass: When the Combine JS and CSS Requests test passes, it means there were no multiple requests for CSS files within 2 seconds of each other, or multiple requests for JavaScript files within 2 seconds of each other. Fail: When the Combine JS and CSS Requests test fails, it means that and fails if there are multiple requests for CSS or JavaScript within 2 seconds. The Results section of this test displays the number of multiple requests for CSS files, and the number of multiple requests for JavaScript files. This test does not have different severity levels. 5.7 Resize Images for Mobile Test This test is conducted by checking to see if there are any images that are 150% larger that the display area specified for them. Page 18 of 57

28 Images that are not correctly sized for mobile can cause extreme delays in rendering. Before content is delivered to a mobile device, it should be resizes to fit the display area Testing strategy and suggestions Ensure that any pages in the app containing image content are exercised, and tested on different sized devices Suggested Best Practices See Resize Images for Mobile for suggested best practices concerning this issue Test Results and Severity The Resize Images for Mobile test has the following possible results. Pass: When the Resize Images for Mobile test passes, it means that there were no images that were 150% larger than the display area specified for them. Fail: When the Resize Images for Mobile test fails, it means that an image was detected that was 150% larger than the display area specified for it. The failure information in the ARO Data Analyzer displays the number of image files that are 150% larger than the area specified for them and contains a table of these files. When you click on any of the files in the table, you will see the Diagnostics Chart with the time position set to the time when that file was sent. This test does not have different severity levels. 5.8 Minify CSS, JS, and HTML Test This test is conducted by checking to see if there are any CSS, JS, or HTML files in the app that could be minified (excess white space removed). Many text files contain excess whitespace to allow for better human coding. These files should be run through a minifier to remove whitespace in order to reduce file size Testing strategy and suggestions There is no specific strategy for this test. Page 19 of 57

29 5.8.2 Suggested Best Practices See Minify CSS, JS, and HTML for suggested best practices concerning this issue Test Results and Severity The Minify CSS, JS, and HTML test has the following possible results. Pass: When the Minify CSS, JS, and HTML test passes, it means that there were no files detected which could have been minified. Fail: When the Minify CSS, JS, and HTML test fails, it means that there are files detected that could have been minified and were not. The Results section of this test displays the number of text files that could be minified and contains a table that lists the file name and shows the percentage of bytes that could be saved by minifying each of the files. When you click on any of the files in the table, you will see the Diagnostics Chart with the time position set to the time when that file was sent. This test does not have different severity levels. 5.9 Use CSS Sprites for Images Test This test is conducted by checking to see if there are groups of small images in the app that can be combined into Sprites, and then rendered with CSS. This will reduce the number of HTTP requests and speed the loading of your app Testing strategy and suggestions Ensure that any pages in the app containing image content are exercised, and tested on different sized devices Suggested Best Practices See Managing Images with CSS Sprites for suggested best practices concerning this issue Test Results and Severity The Managing Images with CSS Sprites test has the following possible results. Pass: When the Use CSS Sprites for Images test passes, it means that there were no groups of images detected that could be combined into sprites. Page 20 of 57

30 Fail: When the Use CSS Sprites for Images test fails, it means that there were groups of images detected that could be combined into sprites. The Results section of this test displays the number of small groups of images that are being downloaded individually in close succession and could be grouped into one image using sprites. The results also display a table showing information about these image files. When you click on any of the files in the table, you will see the Diagnostics Chart with the time position set to the time when that file was sent. This test does not have different severity levels Connection Opening (Self Test) The AT&T ARO Data Analyzer does not conduct a specific test for connection opening. Instead, it displays the following message in the test results that encourage you to analyze any application-initiated data bursts: This test helps ensure that connections are opened explicitly and are grouped together when possible. Some connection startups consist of an input burst, followed by a series of bursts spread out over time which can dramatically slow down the application's response time and waste energy on the device. If you see many application-initiated bursts, consider a transaction manager to group these more closely together Testing Strategy and Suggestions Ensure that your functional tests initiate TCP connections and that you capture and active and an idle trace Suggested Best Practices See Opening Connections for suggested best practices concerning this issue Test Results and Severity This test is informational and is meant to suggest a development best practice. It is not pass, fail, or warning and does not have levels of severity Unnecessary Connections Multiple Simultaneous Connections Test The Unnecessary Connections Multiple Simultaneous Connections test is conducted by checking if there are several bursts in a row that are not user Page 21 of 57

31 initiated in the loaded trace files. For example: 3 bursts in 15 seconds or 4 bursts in 1 minute. This test measures bursts that could be more tightly grouped. Syncing all of your data connections can help reduce the amount of time your application is on the network, reducing the battery drain. That will also make your content appear to load faster to the user Testing Strategy and Suggestions Ensure that your functional tests initiate TCP connections and that you capture and active and an idle trace Suggested Best Practices See Multiple Simultaneous TCP Connections for suggested best practices concerning this issue Test Results and Severity The Unnecessary Connections Multiple Simultaneous Connections test has the following possible results. Pass: When the Multiple Simultaneous Connections test passes, it means that the burst data in the trace appears to be grouped together well. Fail: When the Multiple Simultaneous Connections test fails, it means that there are several bursts in a row that are not user initiated in the loaded trace files. The failure information in the ARO Data Analyzer will display the number of sets of bursts that could be more tightly grouped. When you click on this information, you will be directed to the burst chart in the Diagnostics tab which will show the burst group with the highest number of bursts. This test does not have different severity levels Inefficient Connections Periodic Transfers Test The Inefficient Connections Periodic Transfer test is conducted by checking to see if any periodic connections are detected in the loaded trace files. This test helps ensure that your periodic connections are truly needed for the customer, which if not handled properly could cause excessive power drain. Page 22 of 57

32 Testing Strategy and Suggestions Periodic transfers can be hard to detect during a normal Active User or Passive User trace. The data can be overwhelmed by the other data that is being transmitted. Try an Idle use case trace to see these transfers more easily. Start your trace, launch your application, and if necessary, exercise the feature of the application that you suspect may be creating periodic transfers. Then, allow the application and device to sit without any user input. In some cases, may want your application and device to sit for up to 45 minutes to an hour. It may not be necessary to collect trace video during these traces. For longer traces that are passive, it s recommended that you do not collect trace video. Trace video can be disabled by un-checking the Record Video checkbox when you start the AT&T ARO Data Collector Suggested Best Practices See Periodic Transfers for suggested best practices concerning this issue Test Results and Severity The Inefficient Connections Periodic Transfer test has the following possible results. Pass: When the Inefficient Connections Periodic Transfer test passes, it means that no periodic transfers were detected in this trace. Fail: When the Inefficient Connections Periodic Transfer test fails, it means that a periodic connection is detected in the loaded trace files. The failure information in the ARO Data Analyzer will display the number of different connections that were repeated, the number times they were repeated, and the minimum repeat time. When you click on this information, you will be directed to the TCP flow table in the Diagnostics tab which highlights the TCP connection for the first periodic connection of the shortest period. The following severity levels apply to this test: Sev3: If 3 periodic connections are observed. Sev2: If periodic connections repeat more than 20 times or use >20% of trace energy (as listed in the Statistics tab). Sev1: If periodic connections repeat more than 60 times or use >50% of trace energy (as listed in the Statistics tab). Page 23 of 57