Mobile Application Testing. Services Catalog

Transcription

1 Mobile Application Testing Services Catalog

2 Page 2 Table of Contents Why Mobile Application Testing?... 3 Why Orasi?... 3 Client Success Story... 4 Mobile Application Testing Approach... 5 Mobile Testing Assessment Discovery... 5 Mobile Testing Technology Implementation... 6 Mobile Testing Implementation Road Map Micro Focus Mobile Center... 7 Mobile Testing Implementation Road Map Mobile Labs... 8 Mobile Testing Implementation Road Map Perfecto... 8 Mobile Testing in the Cloud... 9 Private Cloud Hosting Mobile Testing Training... 10

3 Page 3 Why Mobile Application Testing? According to Digital Trends Magazine, only 16% of people will try out an app more than twice if it fails to impress them, and Tech Crunch Magazine reports that 80-90% of all apps are used once and then deleted. And, after surveying more than 3,500, Compuware found that the majority of mobile users have encountered app issues in the recent past citing issues such as freezing, crashing, slow launches, not launching, and just not living up to the users expectations. IT leaders have quickly recognized the unique realities of mobile testing. Mobile testing is essentially software testing in which mobile devices are just new target platforms. So, all existing software quality management processes must be applied. Yet at the same time, mobile testing must be capable of handling multiple devices and operating systems, while also addressing the issues of global coverage, regional differences, and mobilityspecific application tasks, product logistics, and apps procurement. Those who don t wish to be on the wrong side of the above statistics and want to ensure optimum performance and functionality from mobile applications, need an efficient and reliable testing strategy. Here are some questions you may be asking yourself: Am I testing on real devices? Simulators and emulators only mimic real device operation and often are not available for months after the devices are released. Only by testing on real devices running the vendor operating systems can you guarantee your app s end-user experience. And, you ll definitely need the real devices to test advanced sensors and gestures that emulators don t support. Am I testing on real networks? Testing a mobile application through WiFi or local networks is not the same as testing on real carrier networks. Only by testing on real carrier networks can you be sure to catch transmission specifics that might cause your app to run slowly or fail altogether. In many cases, network simulation can also be used to help isolate those errors and reduce your overall cost. Am I testing on enough devices? Not all phones are created equally. Since it s not physically possible to test every available mobile device, a representative sample that best approximates your consumer base should be tested. Some of these devices will be part of your build event cycle and some will likely only be used during manual test cycles. Where can automation help? It would take an army of testers to test every use case on every device. While not every mobile test case can be automated, test automation can help remove the manual bottleneck and free you up to focus on the most critical use cases. And, integrating test automation directly into mobile build cycles helps isolate potential problems early before they cause you to miss an opportunity or lose customers from poor quality. Why Orasi? Mobile application testing is here to stay. You need a mobile strategy, and you need help building that strategy. At Orasi Software, our extensive experience with both traditional and mobile applications has enabled us to effectively translate traditional testing methods to a mobile environment. We provide real devices on live networks to ensure that you re testing under real-world conditions. And, we provide experienced personnel to maximize your mobile testing efforts and automate for continued success.

4 Page 4 Responsiveness to the mobile market is mission-critical and you need a trusted advisor for mobile process and technology. Orasi is a recognized leader in the mobile application testing space and can jump-start your mobile application testing initiatives by delivering professional mentoring through certified consultants with hands-on experience. We supply true end-to-end service offerings for one-stop shopping. Client Success Story Orasi was selected by a major auto manufacturer to perform functional testing on a new mobile app. The app included a variety of remote vehicle control functions, including both vehicle status and vehicle operation. Working with the customer development team, Orasi developed an integrated mobile testing dashboard to execute automated tests across the specified set of mobile devices. This approach, now packaged as the Orasi Mobile Device Validation Service, provides status and performance metrics for each test in the dashboard. Project Overview Orasi automated Kia UVO K900 mobile app on Android and ios Executed scripts on 38 test devices Orasi executed 14 test cases (functional and aesthetic tests) on each device 532 individual validations Orasi delivered custom web dashboard to showcase live results across entire device pool Test App: Kia UVO K900 Platform: Android/iOS Native Build ios: si Android: s Testing Tools: Micro Focus Unified Functional Testing (UFT) Perfecto Mobile with Micro Focus UFT Add-in Steps Taken Created a custom hosted dashboard portal Tested 14 unique cases, including both functional and aesthetic conditions Tested 38 devices (14 ios, 24 Android) Identified seven possible result states for each execution Sample ios Execution Findings Six out of 14 (43%) of ios devices tested had no failures or minor aesthetic failures All ios 8 devices (six) had a failure related to a button in the app not functioning Two ios devices were unable to log in and presented wait a moment to synchronize server Sample Android Execution Findings 20 out of 24 (83%) Android devices passed all tests Two of the failed devices only showed minor aesthetic problems related to improper image and text sizes for their small screen size The only functional test failures in any Android device were related to the find my car functionality on two devices

5 Page 5 Sample Cross-Platform Execution Findings The find my car feature was found to cause failures on both Android and ios Of 36 devices tested in the initial execution, 32 devices completed find my car Of the four devices that failed, the failure was unable to be reproduced on two of the devices, indicating a service failure unrelated to the device or the app itself Two Android devices were shown to have reproducible and consistent failures of this feature (Galaxy Note 2, Galaxy S2) Mobile Application Testing Approach The Orasi mobile testing consultants understand that a company s mobile testing practice must integrate with existing quality assurance (QA) architecture and also recognize the challenges, limitations, and additional functionality of mobile technology. These consultants have years of experience implementing mobile testing solutions, as well as dealing with the issues that come along. At Orasi, we know how expensive and frustrating it can be to introduce a new technology. That s why we focus on taking the best of what you do today and helping you efficiently add the new mobile technologies to it. Our comprehensive approach includes not just the tools but also the customization, training, and mentoring you ll need to get off the ground quickly. And, our consultants are well versed in the related mobile technologies, including many mobile vendor certifications for the related environments. Orasi services are customizable too. We can help you define the right process, select the appropriate tool, implement the tool, and even deliver the mobile testing project. Our end-to-end mobile offerings can be assembled in any combination to meet the unique challenges you re facing. Whether you need longterm onsite assistance or just a couple of days of training, Orasi is here to help! Mobile Testing Assessment Discovery During Orasi s functional mobile testing assessment and road map service, an Orasi mobile testing consultant will work with the customer s mobile testing and QA teams to evaluate existing personnel, processes, and technology to determine how they are currently being leveraged. The deliverable will be a strategy and road map tailored to your business goals and objectives. Mobile Testing Considerations Device Management/Interaction What solution is best for your organization (e.g., local vs. remote, large vs. small)? Manual vs. Automation: What Makes Sense? Some features are unique to mobile devices; can they be automated? What functionality does it make sense to automate? How well can I expect automated testing to work with my app? Which automation tools or techniques should I use? Mobile Device Specific Tests How important are interruption tests to your audience? How does the quality of signal affect your performance? Device Selection or Should I be testing on Blackberry? How can you select the right devices to represent the users of your mobile applications? What platforms/os versions should you be testing on? Real Devices or Emulators? Is there a role for emulators? Should you be performing integration testing on real devices instead of on emulators? Non-Jailbreak vs. Jailbreak Devices? What are the risks associated with testing on jail broken devices?

6 Page 6 Functional Mobile Testing Road Map The mobile testing approach road map is focused on integrating mobile testing into an organization s current QA processes for seamless end-to-end testing, including the GUI, web services, and device layers. The implementation road map will address the following capabilities: Process: current state and recommended desired state of the software quality management (SQM) process Organization: current roles and skills and a recommended organizational structure that supports the recommended desired SQM state Deliverables: current and recommended set of deliverables produced by the SQM process Tools: current and recommended software tools used to support the SQM process and potential elimination of tools Audience: intended/actual audience who is using the target app(s), as well as what devices are being targeted for development and testing Staff Experience: the knowledge, skills and ability of the SQM organization, including the current state of skill sets and recommended skill sets to support the proposed desired state Techniques: the definition, usage, and management of SQM standards and best practices The discovery process typically takes two to three days, depending on the availability of QA personnel. After completion of discovery, the strategy document will be completed in one to two days then presented to and reviewed with the customer to identify priority and timeline(s) based on implementation feasibility. Once the priority and timeline have been agreed upon, the road map document will be created to ensure it meets the organization s objectives. Deliverable: Mobile Testing Implementation Road Map The mobile testing implementation road map is customized based on each customer s needs. It typically addresses the following topics and best practices: Test management strategy Optimal testing strategy Categories/types of testing employed in a typical QA cycle Test automation priorities Automation schedule and milestones Progress and status reporting strategy Roles and responsibilities Mobile Testing Technology Implementation After the completion of stage one, the customer can determine the best way to implement the functional mobile testing road map. Options include: Implementing the road map activities with their own resources Having Orasi resources assist them with selected road map activities Having Orasi implement the road map activities as a turnkey service

7 Page 7 Depending on the customer s decision to move into the implementation with Orasi, stage two will also be used to plan the execution of the road map. Implementation activities can include: Resource and project planning Defining QA processes and procedures Tool implementation and training Manual and automated test case creation Mobile Testing Implementation Road Map Micro Focus Mobile Center Engagement Activities Install Micro Focus Mobile Center Server Connect Micro Focus Mobile Center to the customer s test network Connect customer-supplied Android and Apple mobile devices Set up an account for the customer s Micro Focus Mobile Center administrator Demonstrate to the assigned customer s Micro Focus Mobile Center administrator how to create an administrative account Connect to Micro Focus Mobile Center from User Accounts to Confirm Connectivity Mentor customer on how to connect to Micro Focus Mobile Center, install mobile applications, connect and administer mobile devices, and perform both manual and automated tests Prepare Apps Demonstrate app instrumenting for ios and Android Integrate Micro Focus Mobile Center with existing Micro Focus tools including: Micro Focus UFT Micro Focus Performance Center Micro Focus Sprinter Micro Focus Network Virtualization Automation Tool Mentoring (as time allows) Mentoring in the use of the automation tool to prepare resources for automated scripting framework development Will work with the team to establish a framework approach to maximize reusability and flexibility automated test script development Shoulder-to-shoulder work with the team to kick start automation development Engagement Assumptions and Acknowledgements 1. Mobile devices (that are supported by Micro Focus Mobile Center) for testing have been purchased and are onsite 2. Location with power and network access for Micro Focus Mobile Center server and storing mobile devices 3. Customer has identified user(s) to act as Micro Focus Mobile Center admin 4. Identified Micro Focus Mobile Center admin has availability during installation Deliverables The customer can expect to receive the following deliverables upon delivery of the implementation services: Complete installation of Micro Focus Mobile Center Record of devices under test and review of inventory best practices Review of features and functionality of Micro Focus Mobile Center

8 Page 8 Mobile Testing Implementation Road Map Mobile Labs Designed for enterprise testing teams, Mobile Labs deviceconnect s powerful, web-based management console lets you remotely manage, track and monitor your testing assets. But while the tool itself can take mobile testing to the next level, many organizations need a little help getting started, especially if they re new to mobile testing. At Orasi Software, our extensive experience with automated testing and Mobile Labs deviceconnect enables us to help companies get up and running quickly, ensuring that they get the most out of their new investment. Engagement Activities Install Mobile Labs deviceconnect Unpack and set up deviceconnect, including connecting deviceconnect to your test network Set up and demonstrate how to create an Administrative account for the your deviceconnect administrator Install customer-supplied Android and Apple mobile devices and ensure connectivity Connect to deviceconnect from user accounts to confirm connectivity Initiate manual and automated testing processes Automation tool mentoring (as time allows) Mentoring in the use of the automation tool to prepare resources for automated scripting Framework development (as time allows) Work with the team to establish a framework approach to maximize reusability and flexibility Automated test script development (as time allows) Work with the team to kick start automation development Engagement Assumptions and Acknowledgements 1. Mobile devices (that are supported by Mobile Labs) for testing have been purchased and are onsite 2. A location with power and network access for deviceconnect and storing mobile devices has been identified and prepared 3. Customer has identified user(s) to act as deviceconnect administrator(s) 4. deviceconnect administrator has availability during installation Deliverables The customer can expect to receive the following deliverables upon delivery of the implementation services: Complete installation of deviceconnect Record of devices under test, with review of inventory best practices Review of features and functionality of deviceconnect Mobile Testing Implementation Road Map Perfecto Engagement Activities Install Perfecto MobileCloud client environment Set up an account for the customer s MobileCloud administrator Demonstrate to the assigned customer s MobileCloud administrator how to manage the MobileCloud environment Connect client machines to Perfecto MobileCloud Install automation tools (where required) on client machines

9 Page 9 Mentor clients on how to connect to Perfecto MobileCloud, install mobile applications, connect, and administer mobile devices, then perform both manual and automated tests Connect customer-supplied Android and Apple mobile devices to MobileCloud using LocalLink connection (where required) Automation Tool Mentoring (as time allows) Mentoring in the use of the automation tool to prepare resources for automated scripting framework development Work with the automation team to establish a framework approach to maximize reusability and flexibility Automated test script development Work shoulder-to-shoulder with the automation team to kick start automation development Engagement Assumptions and Acknowledgements 1. Any local mobile devices for testing are supported by Perfecto, have been purchased, and are on-site for use during the engagement 2. Customer has identified user(s) to act as MobileCloud admin 3. Identified MobileCloud admin has availability during installation Deliverables The customer can expect to receive the following deliverables upon delivery of the implementation services: Complete installation of Perfecto MobileCloud environment Record of devices under test and review of inventory best practices Review of features and functionality of Perfecto MobileCloud Mobile Testing in the Cloud In today s enterprise environment, any fully-functional mobile testing automation solution will leverage the robust capabilities of cloud services. Cloud-based automated testing provides a 100 percent web-based Software-as-a-Service (SaaS) platform and an end-to-end mobile application lifecycle management solution. IT units can leverage cloud-oriented mobile automation to enjoy real-time access to hundreds of real mobile phone handsets or tablets devices that are connected to multiple live mobile networks in various geographical locations worldwide. Depending on their functional and security requirements, organizations can utilize public, private, on-premise, or hybrid clouds for automated mobile testing. Functional Testing Focused on customer use cases, functional testing ensures the application works as it should on various devices and networks. Security Testing Addresses the safety of a mobile app, identifying possible security vulnerabilities that could threaten the enterprise environment.

10 Page 10 Performance Testing Focuses on responsiveness and scalability. Monitoring Ensures an application continues to work as required over time. Cloud-based automation can deliver powerful mobile application testing capabilities. Private Cloud Hosting The Orasi Private Cloud Hosting service eliminates the need for customers to purchase, configure, and maintain their own mobile devices. Orasi will install and configure all servers and devices necessary at the Orasi Mobile Hosting facility. Customers then access the environment through a secure VPN connection, enabling both manual and automated testing against real mobile devices. Since it s a private cloud option, all servers and devices are dedicated to the customer and available at any time. There is no security risk of other customers having access to the environment or sharing the devices. The secure VPN connectivity provides a dedicated channel by which any customer representative can access the devices from anywhere in the world. The standard term is for one year and is renewed annually, but there are other term options available. During the term of the service, the equipment and software are dedicated to the customer. At the end of the term, the customer has the option to renew or cancel the subscription. If the customer chooses to renew the subscription, all existing equipment is retained and the device pool is reevaluated. Devices Either Orasi or the customer can provide the mobile devices necessary. Orasi s standard hosting package includes two mobile devices of the customer s choosing for each concurrent license. Additional devices can be added to the hosting package to create a customized library of mobile devices for development and testing. All devices are working, physical, mobile devices not emulators. The customer can specify any combination of manufacturer, model, and operating system version for each device. All devices include WiFi connectivity for access to the internet and can be optionally connected to a mobile carrier SIM contract (e.g., AT&T, Verizon) for voice, text, and data testing. If a customer chooses to cancel the hosting service, Orasi will reset all hardware to factory condition and destroy any backup of customer data. At a customer s request, Orasi can physically destroy the mobile devices to ensure data privacy. If the mobile devices were provided by the customer, they will be returned within 30 days of contract termination. Mobile Testing Training Orasi delivers the class, Implementing Better Testing Practices, to organizations to help them meet today s mobile application testing challenges and provides a more effective and efficient approach to testing. Implementing Better Testing Practices is an accelerated and cost-effective approach focused on the proper use of testing methods, tools, and techniques and is presented with a balance of lecture and hands-on exercises.

11 Page 11 Process Training This one-day, on-site training class is provided for up to 20 students and is delivered by a senior Orasi consultant. The following topics are covered: Mobile Concepts What is different about mobile Architecture complexity Testing types specific to mobile Mobile Testing Challenges Connectivity Native device differences Infrastructure Latency Interoperability Mobile Test Strategy Prioritize what to test Build the plan Device validation Types of test cases to develop Target Audience Students will be exposed to lectures and hands-on workshop exercises and have the ability to ask questions during the session. Subject matter experts learn how to test mobile applications efficiently using best practices Manual testers learn how to accelerate manual test case creation and execution for mobile application projects Materials Provided Student participant guide containing the course slides Student class exercises Topics Covered How to manage the challenges of mobile testing How to apply different mobile test strategies How to create different types of mobile test cases When to use test automation on a mobile project For more information, contact Orasi today Orasi Software, Inc. The information contained herein is subject to change without notice. Orasi is a trademark of Orasi Software, Inc. All other product and company names are trademarks or registered trademarks of their respective owners. Document #: 01SC0515A R1116