Intelligent Mobile App Testing

Transcription

1 White Paper Intelligent Mobile App Testing The field of software functional testing is undergoing a major transformation. What used to be an onerous manual process took a big step forward with the advent of Selenium and other testing tools. But those tools remain heavily developer-centric and are primarily tethered to rigid script-based approaches that do not scale. And now, with the rise of cloud, big data and machine learning, companies need to re-think their testing strategies and technologies yet again. The core concept behind this new approach is that, in this era of agile development and rapid customer response, testing can no longer be an adjunct to the development process: it needs to be an integral and strategic component of the workflow. Functionize is doing just that with the introduction of our ground-breaking scalable autonomous test platform. The driving force behind this platform is Adaptive Event Analysis TM, a patented and proprietary hybrid of algorithms which span supervised, unsupervised, reinforcement learning and computer vision, enabling lightning-fast test creation and execution, selfhealing maintenance, and actionable analytics. Functionize gives you the ability to perform complete, fully-automated, computer-vision and AI-enhanced mobile testing, eliminating the need to maintain an extensive device inventory. Native-emulation, push-button testing on a wide range of mobile and desktop browsers, and it s readily scalable through the Google Compute Engine and GCE-nested virtualization.

2 Table of Contents An Overview of Mobile Testing 3 Mobile Challenges Today 3 Scripting Mobile Tests 3 Existing Tools Require Scripting 4 Scriptless Mobile Testing on Any Device 4 Scaling Possible by Google s Nested Virtualization 5 Case Study of iphone Browser Testing 5 Asynchronous Mobile Testing 7 Testing Landscape View 7 Testing Mouse Hovers for Mobile Apps 8 Big Breakthrough in Mobile App Testing 9

3 An Overview of Mobile Testing Mobile app testing requires an array of test procedures which go beyond the ordinary functional, performance, and load testing of desktop applications. Because mobile device functionality integrates a SIM for cell network interactivity, a diversity of feature and component interactions affect the app operation on a cell phone. The vast diversity of devices, operating systems, display characteristics and browser innovations pose many challenges to testing the functionality of mobile apps. Mobile Challenges Today Mobile app testing involves a long list of procedures which encompass special considerations because of both hardware diversity and network volume. There are more than 400 cell networks worldwide, each with nuances of connection protocol, as well as browser redirects which can affect app function. Functionize can test apps on all network types. Consider this list of unique mobile app testing inclusion which goes beyond ordinary functional testing and is not required of desktop apps: Laboratory testing of simulated networks Device availability for thousands of new devices Diverse mobile OS platforms Application compatibility issues Location testing by network Scripting Mobile Tests Scripting test cases for mobile devices are among the most challenging obstacles for QA engineers because of the radical diversity of devices, display characteristics, and operating systems. A user may run the app under test in the downloaded app, or the phone s native browser. Now, a user looking for app features not available in the mobile version of the app may request the desktop version of the app. Spread these three conditions across thousands of phones and devices, and it s easy to imagine how complicated test scripting can be for mobile apps. To further complicate the matter of mobile app interface testing, there are a plethora of conditions to be tested on mobile devices, including the emerging Internet of Things devices, plus wearable devices, some of which also contain SIMs. The web app as a whole and each operational subunit must be subjected rigorously to a comprehensive test suite in order to ensure a great customer experience. The goal of functional mobile app testing is to demonstrate that the application was constructed such that specification, design, and construction are congruent with expected user experience.

4 Existing Tools Require Scripting Below is a list of mobile testing tools that predominantly require scripting to maintain test cases. While most of these tools represent incremental advances in mobile testing, they are still developer-centric. Appium Calabash Frank ios Monkey Talk Robotium Selendroid KeepItFunctional EarlGrey Scriptless Mobile Testing on Any Device Scripted test creation for mobile apps is a massive developer s task and creates a bottleneck in agile development. Just like the desktop application, Functionize opens your web app in a mobile browser, and observes and learns from every assertion executed in the test case. Functionize logs when web elements move, when menus change, and even learns to recognize dynamic elements to anticipate your software upgrades. With Functionize there is no need to code anomalies with scripts in various IDEs explicitly. Functionize runs a test case on any mobile device browser precisely in the same way that a human tester would do. The Functionize Test Cloud makes possible the complete testing and evaluation of mobile devices running your app without the necessity of running tests on the individual devices themselves. This is because Functionize perfectly simulates the behavior of each device s behavior in the context of your web application. Google s User-Agent Switcher extension is used to support specific mobile devices during the recording phase of test creation. To record a test case for an iphone, just select the iphone from the menu. During test creation, the browser mimics the phone and renders the page correctly as if it is running an iphone mobile layout. This includes the scaling of dimensions to run successfully on the iphone display.

5 (Google Chrome User-Agent Switcher) Scaling Possible by Google s Nested Virtualization Flexible scaling of mobile testing has traditionally been challenging and expensive. Performant, accurate emulation is compute-costly virtualizing a mobile device or desktop PC requires hardware acceleration (e.g., Intel VT-X). And hardwareaccelerated nested virtualization wasn t (or at least, didn t used to be) something you could get from public clouds. Enter Google Cloud. By collaborating with GCE and building on Google s Nested Virtualization, Functionize is able to emulate the CPU and other hardware specifically for testing devices running your web app. Functionize s mobile device testing implements multiple levels of virtualization. We start by implementing the test in a virtual machine and then layer in the virtualization of the specified device s hardware using Google s Nesting Virtualization. Case Study of iphone Browser Testing As mentioned, Functionize testing is not dependent on the browser running in the mobile device, but instead, it is device dependent. For example, if you are running test cases to compare ios 8 and the ipad, Functionize observes substantial differences in the page layouts for the two devices. For this reason, the test cases are recorded based on unique device features such as display size and resolution, so that the correct user experience is rendered and recorded by the Functionize Smart Recorder.

6 The variation among devices leads to the crucial question of how much tolerance or margin there is when testing more than one device with a unique, single assertion. In other words, how do we know when a separate test is required for a unique feature of a given device? It may turn out that different screen sizes result in the same layout; (see diagram below) but if two different screen sizes result in different user experiences, then a separate test case must be recorded. Yahoo.com - Desktop View Yahoo.com - Mobile View

7 From one device to the next, it is primarily the rendering and layout of the device which is targeted for detecting visual rendering failures. In this way, Functionize is capable of running tests on all new hardware up to the minute of deployment and delivery. We continuously track firmware updates to guarantee the precision rendering that you require for the best possible user experience of your web applications. Asynchronous Mobile Testing Looking at Functionize s control panel as you build test cases, you will notice that the browser icons indicate the current state of progress for the respective test cases and that each of these is processed as an individual entity in parallel. This includes the capability to manage hundreds of tests asynchronously on potentially thousands of virtual machines. Testing Landscape View Another important consideration for mobile testing is to consider how the rotation of a screen from portrait to landscape affects the test experience. The orientation of the device can be controlled via settings for the test case on the Functionize UI. Additionally, Functionize allows for configurable testing based on bandwidth experience (e.g. wifi vs. 3G). Throttle Tests by Bandwidth

8 Testing Mouse Hovers for Mobile Apps One of the intelligent features in the Functionize platform is the ability to capture mouse hovers. Ordinarily, this event is captured as a tap event. Functionize actually captures the mouse hover in a mobile browser app. In fact, Functionize automatically recognizes and converts desktop events into their mobile equivalents with no scripting. This ensures the conformity of a tester s experience across both desktop and mobile devices. Big Breakthrough in Mobile App Testing By leveraging Google s new CPU virtualization, Functionize now enables QA teams to dramatically reduce costs, time to test and the pain of maintaining a complex device inventory. Instead of fully testing only certain releases, Functionize automation lets testers verify every software change, and test production sites in realtime to ensure an excellent experience for every user, every client. Functionize heralds a new wave of intelligent testing across the plethora of mobile devices which have long tortured testers with their confusing variety of hardware features. Compatibility issues no longer plague the process. By allowing for parity between testing on desktop and mobile, QA engineers are now free to test any device from a single platform, harnessing the intelligence of Functionize. References Copyright 2018 Functionize, Inc. All Rights Reserved 156 2nd Street San Francisco, CA