A White Paper Analysis from Orasi Software. Selecting Which Mobile Devices to Test

Transcription

1 A White Paper Analysis from Orasi Software Selecting Which Mobile Devices to Test

2 General Device Selection Mobile Application Testing presents several unique challenges. The most formidable of these is supporting the variety of operating systems, their versions, and hardware manufacturers. Today, four out of five devices sold worldwide are Android devices. There are currently seven versions of the Android operating system that have market share; and of the top twenty Android phones, the majority are Samsung Galaxy devices. Based on this information, an organization could assume that performing their android testing on Samsung devices across multiple versions would be sufficient. Unfortunately, there are a number of additional concerns that need to be addressed which make mobile testing a much more complex problem. While it is physically impossible to test every device, to avoid unpleasant surprises organizations should conduct an analysis in order to create a representative sample for each application. As an example, an entirely generic device list might look like this: 4-6 Apple iphone and ipad devices covering a range of IOS versions and carriers 6-12 Android handsets and tablets covering a variety of manufacturers, OS versions, and carriers In the past, it has been recommended that organizations obtain and maintain the latest and most popular versions of Blackberry and Windows phones. With the huge popularity of ios and Android and the decline of Blackberry users, testing on these devices is only recommended if the devices represent a large number of the user base, or the device is company-issued. For consumer applications, a good rule of thumb is that a typical organization should test on roughly 20 different devices in order to cover at least 80 percent of the devices being used by users. To further complicate the challenge, new devices release at the rate of about three to four additional models every quarter. Manufacturer Differences Even when using the same OS and version, most mobile device manufacturers add proprietary elements to the devices. Also, most manufacturers provide unique context-specific software buttons that offer different functions. These context-sensitive features may often differ from device to device, or even release to release. For example, on the next page is a screenshot of the Android Settings app on an LG phone. Similarly, on the right is the same screen taken from a Samsung Android device. 2

3 Market Research For QA organizations with the resources, market research can be a valuable tool to determine which mobile devices their customers are currently using. This can either be a mix of surveys and polls for internal users, or the engagement of marketing firms for larger, external audiences. This is an inaccurate method of determining a user base for a new implementation, but it s a good start when organizations are faced with no historical data to draw upon. Figure 1: Android Settings app on an LG device (left) Figure 2: Android Settings app on a Samsung device (right) You can see that even though the OS (Android) and version (4.1.2) are the same, the screen displayed is significantly different. This can be a serious issue if the testing you are doing relies on visual elements. However, if your testing is based on objects native to the application, e.g. using instrumentation, this will be less of a concern. How to Determine What Devices are Most Important With all the different devices available on the market, it is difficult to select a perfectly representative sample of devices on which to test. Short of randomly selecting a group, there are two primary methods to determine which devices you should be testing. These include market research and detecting mobile devices from your existing online services. Mobile Device Detection Mobile Device Detection is the most reliable and dynamic way of determining what devices your existing audience is using. This can be done through a website or through the application itself. Device detection captures statistics about the OS, the OS version, and the physical device being used to access your application. Solutions range from simply determining whether a user is using a mobile device, to sophisticated systems that can track granular details like the device manufacturer and model. These details are then used to determine the most broadly represented devices among the app s current audience. While this data can t be the sole determinant for selecting a testing device pool, it does add a level of realism to the selection process so that you can make informed decisions. Sample Device Selection Scenarios On the next page are a few scenarios that demonstrate the device selection process for various application types. Of course, each situation you encounter will be unique but these scenarios can provide a starting point for your decision. 3

4 Scenario #1: An app is developed for a single proprietary device This is the easiest scenario to plan for. The testing has a pre-defined scope consisting of a single device. This scenario would result in a short device list created by combining the proprietary device with the OS version that could be installed. Most often, testing must include the most current production OS version and the one prior so only two variations need be considered. Scenario #2: An app is developed for only the ios operating system This scenario is a bit more complicated than the first scenario because, while there is still only a single manufacturer to consider, the device types are multiplied. In addition, knowledge of ios itself might be necessary to determine which specific devices to select. Virtually all Apple devices are supported for the most recent version. However, once a new version is released, Apple generally discontinues support for it on all but the oldest devices. In fact, unlike the Android market, Apple users tend to upgrade almost immediately when a new release is available, so testing older versions of ios is not as critical. For example, when ios 7 was introduced, Apple discontinued support for ios 6 on all but the iphone 3GS and the 4th generation ipod Touch. Given Apple s support policies at that time, a practical device list might include 2-3 iphone models running ios 7, 2-3 ipad models running ios 7 and the two supported devices running ios 6. That means the device list could include 6-8 unique devices in order to test the most popular combinations available in the market at the time. In order to further narrow down this list and focus on only relevant devices, surveys and market research can also be employed. In an example of market research based on an industry, you may discover that in your industry, customers seem to use the iphone 5s more than the iphone 4s. With that information, you could eliminate the iphone 4s as a necessary device to test. Scenario #3: An app is developed for both ios & Android Introducing Android support brings with it many more device versions and OS differences. Devices considered current exist in far more variations, including numerous manufacturers, models, and operation system variants. Of course, the ios device list described above would probably not change unless you employ market research to make adjustments. However, introducing Android into the mix could produce enough variability that it s worthwhile to add dynamic device detection into the process. Without additional custom analysis, it s recommended to select the top 3 Android manufacturers and to select 1-2 representative devices from each. In today s market, for example, that might include Samsung, LG and HTC. Since Samsung devices currently have such a dominant position in the market, you might select several Samsung devices and only a single device from the others. However you define the mix, you ll probably need at least 6-8 devices to represent the various manufacturers and models, including both phones and tablets. In order to test against older versions of Android, you may then have to double this number. Android users don t upgrade as often as their ios counterparts, however, so you ll have to review 4

5 which models make sense to test which Android versions. This is because the manufacturers are generally less likely to provide upgrades for older models and encourage a device swap instead. It is often the case, in fact, that a given Android device can only support a single version of the operating system. As mentioned earlier, dynamic device detection can then be used to eliminate devices from your list that aren t represented in the customer base. For example, an app developer might discover that none of their users are currently using an Android version earlier than 4.2.x. In this case, the developer can then either eliminate devices from their overall list or replace the older devices on the list with newer or more varied counterparts. Conclusion Deciding which mobile devices to test against is a complex and highly subjective process. In the end, though, keep in mind that it depends more on the application consumers than on the hardware available in the market. That is, the device pool should represent the devices that will be used by the target audience. Whenever possible, a blend should be created that includes both market dynamics (e.g., manufacturer market share) and current usage statistics captured through dynamic device detection methods. About Orasi An HP Software Platinum Partner and authorized Support partner, Orasi resells HP s test management and automation solutions, is a leading provider of software testing services, and offers mobile testing, application security, and cloud-based testing and monitoring solutions. For over 12 years, Orasi has helped customers successfully implement and integrate software testing environments to reduce the cost and risk of software failures. Orasi was HP Software s US Solution Partner of the Year in 2011 and 2013, and was Support Partner of the Year for 2009, 2011 and Orasi offers proven solutions for the installation, implementation, configuration, and use of mobile testing technologies; no matter which devices you want to test, we can help! For more information, contact Orasi today. Learn More: Orasi Software, Inc. The information contained herin is subject to change without notice. Orasi is a trademark of Orasi Software, Inc. All other product and company names are trademarks or registered trademarks of their respective owners. Document #: 01WP0414A 5