Let s Test South Africa. Tim Coulter, 2014

Size: px

Start display at page:

Download "Let s Test South Africa. Tim Coulter, 2014"

Melina Barnett
6 years ago
Views:

1 Let s Test South Africa Tim Coulter, 2014 tim@tjc.io

2 Automation is like beating dumb kids with a stick.

3 Text Automation sucks. (for a lot of reasons)

4 Main points My Background Definitions Value - does it have any value? Pitfalls - with examples! Engineering Management and Organizational Cultural Life as an automated tester Is it worth it? My ideal environment

5 Tim Coulter Professional quitter 7 years expl./auto pro Mostly startups Formal training under Kaner Cucumber, rspec, JUnit, JMock, Test::Unit, Selenium, Fitnesse, Kif, jasmine, WATIR it goes on. Homegrown ios framework Homegrown perf framework using JMeter

6 Unpopular opinions

7 Definitions Getting on the same page

8 Definitions: Manual Testing

9 Definitions: Manual Testing Manual testing = Exploratory testing

10 Definitions: Automated Testing Automated testing =! Exploratory If you have to click a button called Run, it s automated testing. (Yes, there s computer-assisted exploratory testing, but that still requires exploration.)

11 Types of Automation That Matter Unit Test (anything running the application through some programmable interface: function call, api call, etc.) GUI Test (anything directly using or simulating the native graphical user interface of the application in order to trigger code execution that produces a result.)

12 Types of Automation That DON T Matter Regression Functional Integration Acceptance Load* These all describe intention, motivation or belief about the purpose of each test, and not necessarily reality.

13 At their core every automated test is a block of arbitrarily executable code that produces at least a binary result, and every framework is a new way of managing these code blocks. For the purposes of this talk, no framework is special. They all do the same thing.

14 Value Do automated tests have value?

15 Yes. Maybe. Kindof. The value of an automated test is inherent in our reason for automating. Core value: change detection. We want tests to fail.

16 Pitfalls: Engineering

17 Pitfalls: Engineering Despite our best efforts, the why isn t communicated (neither is necessity or inherent value) Designing for change detection: not easy, no clear principles, though I have preferences Like code: Poor estimates, tests not factored in The more specific, the more complex, > false positives and/or maintenance cost

18 Public Folders Example eba867e3bcad540a344a b8b7de0a0dea73c40b18

19 Pitfalls: Engineering, con t Integration with third parties Mocking (so, what are you actually testing?) Third-party test environments (lose control) Neither is a great solution Brittleness, especially GUI tests Managing run time

20 Pitfalls: Management & Organizational

21 Pitfalls: Management & Organizational Berlin wall automation: Testers write tests, devs write code Devs are the worst customers Poor communication Testers reactive to test failures, always on cleanup duty (team janitors) Constant reactiveness = no time for improvement Unspoken caste structure: Devs more important

22 Pitfalls: Management & Organizational, con t Ergo: Responsibility mixing. But: not silver bullet How integrated? Are there shared design decisions? Test code is usually a mess, lack of shared goals Short story: My trip to CAST 2012

23 Pitfalls: Management & Organizational, con t Misguided work product # of tests important, or simply the existence of tests All program code must ship with tests Which tests? What strategy? How will it be implemented? Are any extra features required? Development requirements re: test code are not taken into account, adding undue pressure (i.e., tests are code, and all code takes time to write well)

24 Pitfalls: Cultural

25 Pitfalls: Cultural DHH - (unfortunate) mouthpiece of the industry Developers are leading the testing movement Testing synonymous with automation in some development cultures Developers stake identity in process: TDD, BDD, etc. No room for criticism Example:

26 TDD is dead. Long live testing. -DHH We're herding some sacred cows to the slaughter right now. That's painful and bloody. TDD has been so successful that it's interwoven in a lot of programmer identities. TDD is not just what they do, it's who they are. I think that's the direction we're heading. Less emphasis on unit tests, because we're no longer doing test-first as a design practice, and more emphasis on, yes, slow, system tests. Yes, test-first is dead to me. But rather than dance on its grave, I'd rather honor its contributions than linger on the travesties. It marked an important phase in our history, yet it's time to move on. Long live testing.

27 Pitfalls: Cultural, con t Phrases I hate hearing 100% coverage Write a test for every bug Automate [insert amount] % of testing You can automate any test you could run manually

28 Life as an Automated Tester Short answer: Life sucks.

29 Life as an Automated Tester Testers on cleanup duty Hardly interesting code (100 s of short, dumb code blocks. Yay ) Caste system dev focussed Testers duties not included in requirements, undue pressure

30 Life as an Automated Tester, con t Bad, inattentive customers (they re busy too) Management indifference No positive recognition for work only negative when tests fail Automated testers, in my opinion, lack value sync (re: Rob Sab). They re drones.

31 So: Is it worth it? Hint: It depends

32 Is it worth it? Unit testing where devs are 100% responsible: absolutely Anything API related Anywhere where test interface is unlikely to change much Anywhere where the program is unlikely to change much Therefore, GUI testing at least in startups hell no.

33 My Ideal Environment Manual exploratory testing, using tools or automation as needed Developers who manage 100% of the unit testing Little to no GUI testing Organization who values constant investigation

34 Comments Questions Special context? Speak up!

Maja Schreiner. 9th Lean, Agile & Scrum Conference 2017

Maja Schreiner. 9th Lean, Agile & Scrum Conference 2017 Maja Schreiner 9th Lean, Agile & Scrum Conference 2017 Senior Test Master @ Swisscom, Switzerland maja.schreiner@gmail.com testmotion.wordpress.com Twitter: majaschreiner process of executing many different