Michael Yudanin. Chicago Quality Assurance Association November 19, 2013

Size: px

Start display at page:

Download "Michael Yudanin. Chicago Quality Assurance Association November 19, 2013"

Chrystal Spencer
5 years ago
Views:

1 Michael Yudanin Chicago Quality Assurance Association November 19, 2013

2 » Michael Yudanin, Certified Software Quality Engineer (by the American Society for Quality)» In software quality assurance and testing since 1990s» Was involved in testing, process improvement, and management roles that included implementation of quality standards; development of policies and procedures; test planning, design, and execution; management of testing teams and business units; test automation for web, Windows, Unix, databases, mobile, etc.» Worked with large global corporations and federal government: CDC, UPS, Beazer Homes, the Home Depot, the Weather Channel, Bank of America, Spirit Airlines, etc., as well as with small start-up companies» Established Conflair, a QA and testing company, in 2006» Since 2008 active in mobile test automation. Developed RealMobile TM» For more details see Copyright Conflair Inc. All rights reserved. 2

» The multiplicity of platforms Operating systems: ios, Android, BlackBerry, Windows Mobile, Hardware: Apple iphone and ipad, HTC, Motorola, BlackBerry,»

process End-to-end business processes involving mobile front-end and different back-end (web, Windows, mainframe, DB, )» Unique functionality» App stores

4 » The multiplicity of platforms Operating systems: ios, Android, BlackBerry, Windows Mobile, Hardware: Apple iphone and ipad, HTC, Motorola, BlackBerry,» Limited resources Weaker CPU Limited memory» Connectivity modes Network: G3, G4, CDMA, GPRS,, and Wi-Fi Carrier: <a whole lot>» Embeddedness in the business process End-to-end business processes involving mobile front-end and different back-end (web, Windows, mainframe, DB, )» Unique functionality» App stores Location Orientation Barcode scanning Vendor-approved Alternative» Data Entry One has only so many thumbs» and more Copyright Conflair Inc. All rights reserved. 4

5 » Multiple hardware platforms CPU Memory Display size / resolution Keyboard (soft / hard / external) Touch functionality» Multiple operating systems ios versions Android versions Windows Mobile vs. Window Phone BlackBerry» Jailbroken / Rooted Copyright Conflair Inc. All rights reserved. 5

» Relevant to ios and Android platforms» Bringing Unix back to its

to install apps that are not in the Apple / Google store» Legal yet

ios devices remotely NOT necessary for controlling Android devices

6 » Relevant to ios and Android platforms» Bringing Unix back to its original state: removing restrictions and enabling root access» Allows to install apps that are not in the Apple / Google store» Legal yet can violate warranty» As of September 2013, necessary for controlling ios devices remotely NOT necessary for controlling Android devices remotely Copyright Conflair Inc. All rights reserved. 6

7 Platform Code base Coding languages Native App Platform-specific - OS - Hardware type (smartphone/tablet) Separate for each platform >>> mind code generators Different languages for different platforms (usually) Mobile Website Platform-agnostic (usually) One for all platforms (usually) One set of languages for all platforms Dependency None (almost always) Mobile browser Installation Relevant Irrelevant Re-directing Irrelevant Relevant Copyright Conflair Inc. All rights reserved. 7

8 » Not your PC Weaker CPU Little memory» Prone to fail due to memory leaks Endurance/Reliability tests» Cannot install a full-featured test automation tool on a mobile device Copyright Conflair Inc. All rights reserved. 8

» Different network types G3, G4, CDMA, GPRS,,

do you simulate varying reception conditions?

9 » Different network types G3, G4, CDMA, GPRS,, and Wi-Fi Combination of network and Wi-Fi» Different carriers US & Canada Europe Western and Eastern Japan India and China» Reception How do you simulate varying reception conditions? Faraday cage Copyright Conflair Inc. All rights reserved. 9

» Information» Social media» Games» Retail»

Medical Engineering Copyright Conflair Inc.

11 Corporate DB Many mobile apps are merely a front-end for complex business functionality that involves:» Mainframe computers» Database servers» Application servers» Parallel front ends Web Windows Web services Other mobile devices Local WiFi App server Web T1 line Store server Store floor Copyright Conflair Inc. All rights reserved. 11

12 » Gestures & shake-up HW and SW-dependent Conventions?» Location awareness GPS + DB(s)» Orientation Portrait / Landscape Default» App install Store Side-load» Automatic opening of the mobile website version Copyright Conflair Inc. All rights reserved. 12

14 GPS Simulators» Emit signals that mimic live GPS signals» Allow full control over the signal» Can simulate GPS alternatives, e.g.: GLONASS, BeiDou, Galileo Copyright Conflair Inc. All rights reserved. 14

15 » Factors to consider: Types of barcode Fonts Paper Lighting + Research what are the expected lighting conditions + Can be more than one Distance + What is the expected distance from the barcode? + Are there any physical limitations? Copyright Conflair Inc. All rights reserved. 15

16 » Factors to consider: Color Texture Paper Lighting Distance Ink + What is the expected distance from the check? + Are there any physical limitations? + Color + Thickness + Texture Handwriting / typing Copyright Conflair Inc. All rights reserved.

17 » Apple App Store Cydia» Android Google Play (former Android Market) Side-load» Microsoft Windows Store (former Windows Phone Marketplace)» Side load Installing native apps from other sources Copyright Conflair Inc. All rights reserved. 17

» App opening time When not in memory When pre-loaded

» Reliability tests Important to discover memory leaks Outcome frequently depends on the device HW and OS flavor phone holder USB cord to PC on/off switch power

19 » Reliability tests Important to discover memory leaks Outcome frequently depends on the device HW and OS flavor phone holder USB cord to PC on/off switch power status indicator Automation that can access a device that is being physically manipulated Copyright Conflair Inc. All rights reserved. 19

20 » Typing on a smartphone keyboard, hard or soft, takes 2-5 times longer than on a computer keyboard.» No copy-paste from test data files is available. Copyright Conflair Inc. All rights reserved. 20

21 » Hardware compatibility Apple: iphone versions & ipad versions Android: device and tablet versions» OS compatibility Apple: ios versions Android: OS versions and flavors» Carrier compatibility» Mobile Website Does it open automatically on all target platforms? Copyright Conflair Inc. All rights reserved. 21

22 » Data Exchange Exchanging data with the app server, DB server, Data upload after off-line» Invoking functionality Notifications: tray, pop-up, update Real-time messaging Video streaming» App updates Copyright Conflair Inc. All rights reserved. 22

» Is your system in production working properly

Production Monitoring Dashboard Run apps at different

23 » Is your system in production working properly everywhere it is supposed to work? Opening time Main functionality Transaction time Production Monitoring Dashboard Report results to the Production Monitoring Dashboard Run apps at different locations every X minutes Copyright Conflair Inc. All rights reserved. 23

» All the regular reasons plus» Data entry Significant time gap between devices and PC» Time-to-market Increased need for regression testing» Exposure Production

24 » All the regular reasons plus» Data entry Significant time gap between devices and PC» Time-to-market Increased need for regression testing» Exposure Production monitoring» Memory leaks Mobile devices are quicker to fail little memory Reliability tests Copyright Conflair Inc. All rights reserved. 24

THE ULTIMATE CHALLENGE OF MOBILE TEST AUTOMATION 1 2 3 Automating mobile apps and websites as we automate PC applications and websites: All the functionality - Clicking buttons - Entering data -

25 THE ULTIMATE CHALLENGE OF MOBILE TEST AUTOMATION Automating mobile apps and websites as we automate PC applications and websites: All the functionality - Clicking buttons - Entering data - Validating images - Verifying text - Scanning barcodes - Imitating low reception - on real devices while keeping them intact in one script Copyright Conflair Inc. All rights reserved. 25

26 1. Technology i. Platform support + Mobile + Other ii. Device operations + Jailbreaking / rooting + Hardware modification + Instrumentation Connectivity 2. Vendor i. Size ii. History iii. Location (legal) 3. Cost of ownership i. Licensing ii. iii. iv. Other software to purchase / rent + Automation tools + Remote control + Hardware to purchase / rent Script development effort v. Skill acquisition + Training + Hiring Copyright Conflair Inc. All rights reserved. 26

28 » A piece of software installed on the PC that emulates the device. PRO 1. Easy to install and manipulate 1. Not realistic CONTRA 2. Not always up to date on the latest patches of the OS Copyright Conflair Inc. All rights reserved. 28

» Hard-wiring the device to allow remote control. Internet PRO 1. Device hardware utilized 1. Not realistic 2. Tools usually maintained by at least midsize companies CONTRA 2.

29 » Hard-wiring the device to allow remote control. Internet PRO 1. Device hardware utilized 1. Not realistic 2. Tools usually maintained by at least midsize companies CONTRA 2. Might take significant time to add newly released devices to the lab 3. Highly complex hardware and software Copyright Conflair Inc. All rights reserved. 29

» Instrumentation: adding special hookups to the application under test to control it by signals from a computer WiFi / USB / other conneciton Tool PRO 1. Real devices utilized 1. AUT changed 2.

30 » Instrumentation: adding special hookups to the application under test to control it by signals from a computer WiFi / USB / other conneciton Tool PRO 1. Real devices utilized 1. AUT changed 2. Little load on device s memory and CPU 3. Communication-agnostic >>> depends on the tool used CONTRA 2. Security / configuration management risks 3. Cannot open AUT automatically 4. Cannot work with other applications, e.g., Settings Copyright Conflair Inc. All rights reserved. 30

» Control the device remotely; jailbreak if needed» Decoupling

conneciton Tool PRO CONTRA 1. Real devices utilized 1.

Little load on device s memory and CPU 3.

31 » Control the device remotely; jailbreak if needed» Decoupling automation tools, remote control, and the AUT WiFi / USB / other conneciton Tool PRO CONTRA 1. Real devices utilized 1. Jailbreaking required for ios 2. Little load on device s memory and CPU 3. Communication-agnostic >>> depends on the tool used 4. Can use common automation tools 2. Remote control software required usually platform-specific 3. Usually cannot simulate gestures Copyright Conflair Inc. All rights reserved. 31

» Tools dedicated to automating tests for mobile platforms Different approaches: HW modification, instrumentation, remote control Some have unique scripting languages Some are add-ons to common tools

32 » Tools dedicated to automating tests for mobile platforms Different approaches: HW modification, instrumentation, remote control Some have unique scripting languages Some are add-ons to common tools Usually support 1-2 platforms Accommodating new platforms requires changing to the tool All are supported by small or midsize vendors An add-on to a common tool is akin to a stand-alone tool with added dependency.! Copyright Conflair Inc. All rights reserved. 32

» Mobile test automation as a method: utilizing common tools in a way that allows automating mobile interfaces» Usually use jailbroken ios devices» Use common remote control applications»

33 » Mobile test automation as a method: utilizing common tools in a way that allows automating mobile interfaces» Usually use jailbroken ios devices» Use common remote control applications» Platform-agnostic Utilizes multiple remote control apps» Communication-agnostic» Can support all non-mobile platforms that the utilized tool supports Copyright Conflair Inc. All rights reserved. 33

You will find videos that record demos of using common tools to automate tests for mobile devices at: https://www.youtube.

(1) generates barcode (4) validates scan Automation tool (2) prints barcode (3) scans barcode by

37 » The task of mobile test automation is to address the challenges Multiple platforms Limited resources Variety of connectivity modes Unique functionality while working with real, intact devices» None of the currently available methods can do that 100%» Each enterprise should compile a set of criteria in order to select the best method for its needs Copyright Conflair Inc. All rights reserved. 37

Michael Yudanin, CEO Conflair (678)467-1464 michael.

38 Michael Yudanin, CEO Conflair (678) Conflair Inc 38