Better late than never

Size: px

Start display at page:

Download "Better late than never"

Opal Lester
5 years ago
Views:

1 Better late than never Integrating Selenium after the fact R. Tyler Croy

2 Hello and thanks for coming. I'm R. Tyler Croy and I'm going to talk a bit this evening about the integrating Selenium and browser testing after you've already built out a mature web application Before I get to the good stuff, first let me explain who I am and why I'm qualified :)

3 I work here

4 Hacking with Ruby

5 Most of my work is in Ruby, while Lookout's flagship product is an application for the Android platform, the vast majority of our code is all server-side Ruby code. For my team specifically, we're responsible for Lookout's web presence, which is a lot of Ruby on Rails (for better or worst)

6 I am not in QA

7 (so why do I care?)

8 Quality is important

9 Quality is important for a lot of reasons, in Lookout's case a fundamental part of user trust is reliability in my opinion. If a user loses their phone, visits mylookout.com to try to find the phone and our site breaks, they're going to be pretty pissed. Automating the quality of the app is also pretty important IMHO to ensuring that we as a team can move forward and focus on building new things instead of trying to make sure that all the old stuff doesn't fall over

11 Talk about the "state of Lookout" when this project started around a year ago

12 Challenges

13 Browser test suite

14 Browser testing infrastructure

15 This is a picture of our browser testing infrastructure when we started We had absolutely nothing in place, this means we had no machines to run browsers, our build machines were a Fortunately we did have some amount of automation hooks exposed for creating accounts on a staging server and manipulating/scaffolding up "state" on a server

16 Unit Test-Driven Team

17 The team, was very familiar with using TDD. Not so much with ATDD. Even so, the tests that were typically being written were almost *entirely* unit tests, with a thin veneer of integration/high-level tests being written Need to mention the difference between unit tests (for developers) and acceptance testing (from the user perspective

18 Lack of experience

19 We also had a pretty major lack of experience in browser or higher-level automated acceptance testing throughout the company

20 Directionless

21 Because of this lack of experience we had a lot of uncertainty and doubt with regards to the right direction to start to build a test suite and testing infrastructure for. I personally had some experience from working at Slide on this very kind of project, which was ultimately an abyssmal failure. We used Windmill and created very brittle tests, only written by QA engineers, tied the tests very close to the page driver itself and ended up writing very "unit-test-style" methods/test cases such as "def test_page_does_display_alert_when_clicking_thing" with tens of lines of browser calls in them

23 So that's where we started from, it was scary.

25 I'm going to try to avoid being overly prescriptive, I'm certainly opinionated but I just want to make sure I focus more on what has and has not worked for me at Lookout and a bit at the failed project at Slide

27 QA should not own selenium testing

29 Selenium testing should be moved as much into the app developer workflow as possible.

31 Binding together the starting of a local server running the application, with the running of the tests is a *bad idea*. This will really hurt your ability to run many tests in parallel, but also can increase overhead for every single test case, especially as tasks such as rebuilding the DB, or factorying up data grows.

33 It is a *good idea* to make sure running against a local test server works seamlessly. In fact, ensuring that by default tests run against a local server is fine for development, but for CI and full test runs, it's important that the tests are capable of hitting remote staging or test environments.

35 Use the exact same tool that you write your unit test code in

36 class PurchaseTest include SeleniumHelper def test_valid_credit_card_purchase #... end def test_fat_fingered_credit_card_information user = create_user driver.nagivate.to('/purchase') cc_elem = driver.find_element('input#credit_card') cc_elem.send_keys(8008) button = driver.find_element('input#purchase') button.submit end def test_d2p_for_already_existing_account #... end end

37 The tool isn't designed for the job, it also encourages developers and testers to write terrible tests because they're not breaking out of the scope of a unit test and thinking about things from the "outside"

39 Use Selenese. Does everybody here know what Selenese is? It's an "interesting" langauge that the Selenium IDE records into, that is totally brittle, unmaintainable and write-once read never for your development and QE team Here's one of the shortest examples I could find in our git log of some terrible Selenese that was created long ago

42 Use tooling better suited for acceptance or integration testing, such as Cucumber, JBehave, etc

44 Bad Idea: Using Cucumber *incorrectly* is almost as bad as not using Cucumber at all.

45 Scenario: Given I When I And I And I When I Then I Then I And I Free users can locate and scream am a free user visit the login page enter my password enter my click on the "Login" button click on the "Missing Device" link should see the "Locate" link should see the "Scream" link

46 There's literally no benefit to using Cucumber if this is how you're writing scenarios. This just becomes another layer of useless indirection Mention the issues with internationalization

48 Using Cucumber is a great idea, in my opinion, for creating a focus on Acceptance Test Driven development. The structure of it, and the separation from the code itself can really help developers think and communicate in the abstract for a given task or feature.

49 Scenario: Free users should have locate/scream capability Users that are either in trial or in the free account state should always be able to locate and scream from the non mobile version of the web site Given When Then And I I I I am a logged in free user visit the "Missing Device" page should be able to locate my phone should be able to scream my phone

51 Use the Selenium WebDriver API directly

52 @driver.navigate.to(url_for(:controller => 'user', :action => 'login)) _field => 'login[user]') # Make sure we found it! _field.should_not be_nil _field.send_keys(user. ) pass_field => 'login[password']) pass_field.should_not be_nil pass_field.send_keys(user.password) button => 'login') button.should_not be_nil button.submit

54 Use Capybara, if you're a Ruby developer

55 # Visit our Login page visit url_for(:controller => 'user', :action => 'login') # Log in! fill_in "login[user]", :with => user. fill_in "login[password]", :with => user.password click_on 'Log in'

57 Build an automation library or layer that the Cucumber step definitions can talk to.

58 Given /^I am a logged in (free pro) user$/ do type # Call our automation API user = create_user(type.to_sym) # Visit our Login page visit url_for(:controller => 'user', :action => 'login') # Log in! fill_in "login[user]", :with => user. fill_in "login[password]", :with => user.password click_on 'Log in' end

59 Given /^I am a logged in (free pro) user$/ do type # Call our automation API user = create_user(type.to_sym) # Call our automation API to login login_with(user) end

61 Re-use step definitions within other step definitions

62 Given /^I am a logged in free user$/ do Given %{I am a free user} Given %{I am logged in to the site} end

64 Spent time and money setting up a local selenium grid or other testing infrastructure before getting the company bought into browser testing

66 Use hosted or managed services such as Sauce Labs to get started with testing multiple browser versions very quickly

68 Run all automated tests against a single host.

70 Set up multiple staging or testing environments, ideally those in some sort of elastic cloud such as EC2, OpenStack, etc

72 Save test-writing for the absolute last step before deployment, putting a lot of pressure on the QA team but also incentivizing taking shortcuts and not writing tests at all! Tests written as an "after thought" are bound to be more hacky and can not help weed out product or UX issues early enough to stop bugs before they happen

74 Start writing scenarios, or outlining acceptance tests at the top of the "product funnel" to make sure that the team can agree/understand what's being built and why.

76 Rely on Selenium to test/validate all potential behaviors of your front-end code. It's much more time-efficient to write unit tests for JS in Jasmine than running the appropriate Selenium test cases to exercise varying invocations of JavaScript functions

78 Write unit-tests for your JavaScript code using Jasmine, QUnit, etc to ensure that the front-end application works well.

81 Integrating Selenium testing with CI is a great idea, and very important to ensuring the success of the browser testing project and adoption at the company. Talk a bit about the Jenkins set up for Selenium at Lookout

83 The Selenium work at Lookout is far from perfect, and far from complete. We rapidly back-filled our test coverage which was a mixed blessing. We got a lot of coverage fast, but it varies in quality of test from feature to feature and in some cases browser to browser. Where we have had the best results in our tests have been in incorporating them as much as possible with the actual development of the product feature itself.

84 Questions?

85 Thanks

Chrome if I want to. What that should do, is have my specifications run against four different instances of Chrome, in parallel.

Chrome if I want to. What that should do, is have my specifications run against four different instances of Chrome, in parallel. Hi. I'm Prateek Baheti. I'm a developer at ThoughtWorks. I'm currently the tech lead on Mingle, which is a project management tool that ThoughtWorks builds. I work in Balor, which is where India's best