Ingredients Nokia 2

Size: px

Start display at page:

Download "Ingredients Nokia 2"

Harry Crawford
5 years ago
Views:

1 Experiences from GUI Test Automation in Agile Development Ville Ilvonen Principal Engineer, Tools Development, MeeGo Computers, Nokia September 23rd 2010 Seminar on Model-Based Testing of Smartphone Applications 2010 Nokia. 1

2 Ingredients Model-Based Testing project funded by Finnish tax payers and companies (Thanks TEKES et al.) Relatively large agile organization in Nokia Continuous integration system to run automated tests Bunch of developers and testers to pilot project tool set driven by Maemo 6 and smartphone R&D. Experimental GUI test automation tool for Qt World-class expertise on MBT from TUT 2010 Nokia 2

3 Recipe Mix all ingredients in good proportions with values from agile manifesto and serve chilled. Enjoy with your collegues Nokia 3

4 Quotes from the kitchen Ambitious target of our pilot would be to improve MBT awareness. We d also like to deploy practices and toolset in agile development organization. Maybe also execute those test models in continuous integration system let s say over night using path algorithm of choice. Yours Truly Here s flow diagram and architecture of our CI system. We run huge amounts of traditional tests in it. Isn t it impressive? Yours Truly To do modeling properly we d like to have full UI specifications for this unpublished product, please. TUT project personnel Disclaimer: Author didn t record any discussions so quotes are not perfectly accurate 2010 Nokia 4

5 Ideal Catering Proposal Courtesy of Mika Katara et al Nokia 5

6 Some Activities in Agile Kitchen (+ lots of other roles not described here) Developer Works closely with UI designers, architects and test engineers in scrum team. Writes code and tests based on user-stories. Splits stories to tasks in sprint planning. Sprints end to deliveries to product release content. Follows Definition of Done. The most advanced ones design software using test driven development (TDD). UI Designer Creates UI designs and specifications. Provides input for developers. Provides feedback based on delivered content. Test Engineer Tests with developers during sprint or tests delivered users stories from release content. Finds and reports bugs. Test Designer Designs test asset libraries to cover testing of delivered features Some of the GUI tests end up automated based on planned utilization Test Manager Decide which tests to run over given time slot to find bugs and reduce risk Run development of new features in sprints (2 weeks). Get feedback from testing and users, try concepts and improve design. Embrace change. Change is good Nokia 6

7 Quotes from the kitchen - Continued Model-based testing, that sounds interesting but we have these test asset libraries which contain huge amounts of tests. How do you describe run rates and pass rates with models? We need to keep operations running. Test Designers and Test Managers These test modeling tools work pretty well but they have some rough edges from usability point of view. Piloting Developers Models that map states and transitions to detailed application components for interaction and verification are pretty tough to maintain when application changes, some framework library changes and test tool also changes. That s the case with traditional tests as well but models kind of add a layer on top of that. Piloting Developers Formalism in models is an over-kill. Piloting Test Engineer I don t mind formalism but maintenance of models during sprints is tough. Piloting Developer Training was useful, very interesting with good hands-on part. Nice to try with products we work with. Many participants after training provided by TUT researches Disclaimer: Author didn t record any discussions so quotes are not perfectly accurate 2010 Nokia 7

How about MBT fast food instead of French Menu?

with browser X 1. Open browser X 2. Type URL 3.

Close browser Test flash Verify that flash works 4.

HTML Open Browser X Type URL Verify Flash Close

say 150 use cases how can I report test coverage and

8 How about MBT fast food instead of French Menu? Examples of tests in asset library Test plain HTML with browser X 1. Open browser X 2. Type URL 3. Verify plain HTML content 4. Close browser Test flash with browser X 1. Open browser X 2. Type URL 3. Verify that flash works 4. Close browser Test SVG functionality with browser X 1. Open browser X 2. Type URL 3. Verify that SVG content is rendered correctly 4. Close browser Same tests in example model Verify plain HTML Open Browser X Type URL Verify Flash Close browser Verify SVG If I have a model which has let s say 150 use cases how can I report test coverage and where the bugs are? 2010 Nokia 8

9 New Menu Customization of ideal catering proposal per organization but keep it interesting from research point of view. More user-friendly notation and toolset with manual testing support for GUI testing. Realization that majority of GUI testing during product development is not automated but manual. Manual adapts to change very fast. GUI automation is tempting but making it affordable with embraced change is not easy. Test managers expect practical model-based-test reporting solutions with executive summaries Nokia 9

10 Best Tastes Targets? Got some, lost some. Toolset provided by project is fairly robust but not a product. Possible to convince suits to finance further development? Maybe, doesn t look hopeless. One of piloting developers already wrote MBT-adapter to simpler GUI testing tool (Thanks Riku). SW engineering project group at TUT took challenge to create tool for manual test modeling. Let s see what comes from that. Applying MBT to agile RD is completely different from testing of released products. As expected. Discussion to continue project already started. Discussed here? I d love to see this catered with MeeGo Nokia 10

11 References and other links of interest TEMA toolset - NOTE - Internal version was used during the pilot TDriver tool - TDriver for Documentation MeeGo - On MBT adapter for QtUITest - On Maemo 6 and MeeGo - TUT SW Engineer Project (OHJ-3500) - Project proposal 6 (only in Finnish) One group selected the project. Option for MeeGo Award. Info on automated testing system targeted for use in pilot (Proposal) 2010 Nokia 11

CONFERENCE PROCEEDINGS QUALITY CONFERENCE. Conference Paper Excerpt from the 28TH ANNUAL SOFTWARE. October 18th 19th, 2010

CONFERENCE PROCEEDINGS QUALITY CONFERENCE. Conference Paper Excerpt from the 28TH ANNUAL SOFTWARE. October 18th 19th, 2010 PACIFIC NW 28TH ANNUAL SOFTWARE QUALITY CONFERENCE October 18th 19th, 2010 Conference Paper Excerpt from the CONFERENCE PROCEEDINGS Permission to copy, without fee, all or part of this material, except