INTUITEST Using property-based testing to automate test case generation and diagnosis in web-based graphical user interfaces

Transcription

1 October 19 th 2015 Sophia-Antipolis (France) Laura M. Castro Universidade da Coruña INTUITEST 2015 Using property-based testing to automate test case generation and diagnosis in web-based graphical user interfaces joint work with Clara Benac, Henrique Ferreiro, Macías López, Miguel Ángel Francisco

2 Motivation We wish we could auto-magically test at the GUI level, don't we? 2

3 Motivation We wish we could auto-magically test at the GUI level, don't we? It would be like having our users test our product, only without them being the ones facing crashes and other undesired behaviour 3

4 Motivation We wish we could auto-magically test at the GUI level, don't we? Hiring human alpha/beta-testers is tricky Expensive Difficult to communicate Difficult to reproduce Not exhaustive 4

5 Motivation We wish we could auto-magically test at the GUI level, don't we? Automated monkey testing is tricky too Cheap, but inefficient Difficult to debug Not exhaustive 5

6 Motivation We wish we could auto-magically test at the GUI level, don't we? Automated testing is challenging Models are difficult to write Complete models are unfeasible 6

7 Motivation What if we smarted-up monkey testing with property-based testing so that testing would be automated, cheaper, easier to reproduce, more exhaustive, more effective, easier to model? 7

8 The idea To test at the GUI level, we want to... interact in a random manner, performing any action available to the user interact in a series of pattern-steps, when specific interaction-patterns are present let's combine the two! 8

9 The idea To test at the GUI level, we want to... interact in a random manner, performing any action available to the user interact in a series of pattern-steps, when specific interaction-patterns are present let's combine the two! black-box system-level validation 9

10 The idea To test at the GUI level, we want to... interact in a random manner, performing any action available to the user interact in a series of pattern-steps, when specific interaction-patterns are present let's combine the two! black-box system-level validation web-based 10

11 The idea To test at the GUI level, we want to... interact in a random manner, performing any action available to the user interact in a series of pattern-steps, when specific interaction-patterns are present let's combine the two! black-box system-level i.e. follow link click button input text web-based validation 11

12 The idea The random aspect is easy to model, to automate introduces variability into tests that escapes preconceptions and assumptions The specific interactions are easier to model, since they focus on a specific task or subset of functionality allow for more exhaustive testing 12

13 The run() :SUTmodel run() :WebUImodel :WebUIactions :UIframework :WebdrvSession idea setup() setup() test generation test execution get_actions() get_links() activate_element() click_element() end test execution teardown() teardown() 13

14 The run() :SUTmodel run() :WebUImodel :WebUIactions :UIframework :WebdrvSession idea setup() setup() test generation test execution get_actions() get_links() the random aspect: whatever we find on the GUI activate_element() click_element() end test execution teardown() teardown() 14

15 The run() :SUTmodel run() :WebUImodel :WebUIactions :UIframework :WebdrvSession idea setup() setup() test generation test execution the specific interactions: hand over to specific model get_actions() activate_element() get_links() click_element() end test execution teardown() teardown() 15

16 The run() :SUTmodel run() :WebUImodel :WebUIactions :UIframework :WebdrvSession idea setup() setup() test generation test execution the specific interactions: hand over to specific model get_actions() activate_element() get_links() click_element() end test execution QuickCheck stateful models teardown() teardown() 16

17 The implementation WebdrvSession «uses» EQC dynamic cluster WebUImodel strategy WebUIactions +run() +setup() +teardown() -prop_webui(m : webui_model) -setup(m : webui_model) -teardown(m : webui_model) #run(m : webui_model) SUTmodel +run() +setup() +teardown() call setup(m) run quick-check tests call teardown(m) start webdriver session call m.setup() call m.teardown() stop webdriver session call prop_webui(m) super.run(this) Erlang behaviours set base URL for webdriver session +common_actions() +specific_actions() UIframework +specific_actions() 17

18 The validation VoDKATV pilot study 18

19 The validation VoDKATV pilot study 19

20 The validation VoDKATV pilot study Web-based GUI for configuration and administration 36% of the GUI involves form-like interactions 48% of the GUI involves table-like interactions 20

21 The validation VoDKATV pilot study Web-based GUI for configuration and administration 36% of the GUI involves form-like interactions 48% of the GUI involves table-like interactions easy-to-write reusable models 21

22 The validation VoDKATV pilot study Form-like interaction 22

23 The validation VoDKATV pilot study Table-like interaction 23

24 The validation VoDKATV pilot study Traditional GUI-level is manual, ~1h per functionality/iteration, poor regression testing, no negative testing Three bugs in the last 8 months vs. three bugs in one week of testing with new method & tool Test model reuse (migration to new admin GUI) Cheaper, unattended, easy to reproduce 24

25 Wrap-up We have designed a new approach to GUI-level testing PBT-based, enables automated test case generation, execution and diagnosis allows for test-effort reuse and liberates from test-case design We have implemented it into a testing framework combines a model for random browsing with stateful PBT models for common interaction scenarios We have validated it in an industrial system we found several bugs that have slipped into production after more traditional testing 25

26 Wrap-up Framework available at Thanks for your attention! Questions? 26