The Power of Unit Testing and it s impact on your business Ashish Kumar Vice President, Engineering Agitar Software, 2006 1
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 2
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 3
Goals of Software Development Organizations Release software that works, meeting the business need for new functionality, when it s needed, at the budgeted cost Works Meets the need Delivers functionality On Time Within budget 4
What Do They Need for Better Results? The organization as a whole More free cycles to rapidly deliver new functionality Less drag from fragile legacy applications Software developers Easier ways to create code that is right and which can be changed later without the fear of breaking something Quality Assurance and integration teams Clean code that s easy to integrate and test at the system level Managers and team leaders Visibility, with objective data, into the project s real status 5
Unit Testing Helps, a Lot Excellent Good OK Poor Very Poor Code Quality With Unit Testing Continuous Feedback Without Unit Testing Done Sooner!? Fail / Cancel No Feedback Development with Unit Testing Time Integration & System Testing by QA Source: The Software Development Paradox, Alberto Savoia, 2004 6
Why Stick with a Worst Practice? Test Bugs Out Build Quality In Typical software development process Give QA the main responsibility for quality Test late Deliver, then fix problems Accept limited visibility Mature world-class manufacturing process Hold everyone responsible for quality Test early and often Stop the assembly line Measure objectively 7
To Get It Right: Prevention Saves Time and Money $16,000 85% % Defects introduced in this phase Percentage of Bugs $1000 % Defects found in this phase $ Cost to repair defect in this phase $25 $100 $250 Coding Unit Test Function Test System Test After Release Source: Applied Software Measurement, Capers Jones, 1996 8
To Keep It Right: Detect Changes in Code Behavior Change Code A Impacts Code B Continuous Integration Sensors show the impact of code changes High test density brings confidence and agility Continuous integration & test brings the best impact Impacts Code C 9
Unit Testing: It s Great for QA too System Testing GUI Testing Functional Testing Developer Testing Stress Testing Scenario Testing Unit Testing delivers much better code to QA lets QA focus on true system testing cuts QA time, for faster release cycle 10
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 11
Legacy Code (now in Java!) Code without tests is bad code. It doesn t matter how well written it is; it doesn t matter how pretty or objectoriented or well-encapsulated it is. With tests, we can change the behavior of our code quickly and verifiably. Without them we really don t know if our code is getting better or worse. Michael Feathers, Working Effectively with Legacy Code 12
The practical reality Complex Software New Feature More Bugs Fix Bugs More Complex New Feature More Bugs 13
Unit Testing can be hard Foo.java Foo.java FooTest.java FooTest.java FooTest.java FooTest.java FooTest.java FooTest.java Test Code >> Code Under Test 14
Unit Testing can be hard = Class I know intimately (author) C15 = Class I know and understand quite well C13 C14 = Class I understand somewhat = Huh? C9 C10 C11 C12 C1 C2 C3 C4 C5 C6 C7 C8 C9 C11 C1 C2 C6 C5 C7 15
Testing is hard, some data 8% release software to beta sites without testing 83% of their developers don't like to test code. 53% don't like to test their own code because they find it tedious. 30% don't like to test because they find testing tools inadequate. (Source Dr. Bertrand Meyer s talk at Stanford - from a survey of 240 companies in N. America and Europe) 16
Testing is finally getting some respect from developers Agile software development and XP are gaining in popularity Startups as well as large organizations ISVs as well as IT Unit testing is a core practice of Agile methodologies 17
Out of the Dark Ages and into the Age of Paradox The Dark Ages ( just a few years ago) Developers writing tests? You ve got to be kidding. Testing is for the QA folks. The Age of Paradox (today) Theoretical consensus Unit (developer) testing is a great idea. Everyone should do it. Practical reality Developer testing is still only practiced by a minority Testing is a low status activity 18
The Unit Testing paradox Too complex to test Too risky to simplify 19
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 20
The Future of Unit Testing Scenario 1 Back to the dark ages Long term defeat Scenario 2 Unit testing is here to stay, but practiced by a small minority of organizations and developers Minor Victory Scenario 3 Unit testing is practiced by a majority of software development organizations Major Victory 21
Cautious Optimism Many indications that unit testing is here to stay Web Searches* extreme programming - ~2M results unit testing OR developer testing - ~1.8M results junit + java - ~1.5M results All major IDEs have JUnit support Number of books and articles on developer / unit testing Unit testing is a popular topic at developer conferences Growing ecosystem of open source and commercial testing tools and frameworks aimed at developers * Based on Google Search Results in 2/2006 22
So what do we think? 23
Testivus - Tests for the rest of us The Developer Testing Gene T1 (testing resistant), T2 (testing susceptible), T3 (test infected) T3 < T1 + T2 The Testivus Manifesto Less testing dogma, more testing karma Any tests are better than no tests Testing beats debugging Test first, during, or after whatever works best for you If a technique, metric, or tool, gives you more or better tests use it Source: Alberto Savoia at an Artima Blog 24
Developer Testing Master? In a field with General Practitioners, with a growing body of knowledge, eventually specialists emerge Today specialist QA organizations and tools are a given But this is a relatively recent phenomenon Used to be If you can bring it up, it s probably ok Is it time for a Developer Testing Master? A specialist in Unit Testing that gets development teams started NOT a replacement for developers doing testing Tracks unit testing metrics and failure reports Keeps management and the team apprised on testing focus areas and progress Source: Alberto Savoia at an Artima Blog 25
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 26
What Agitar does Practices that go with Unit Testing (taken to the extreme) extreme Continuous Integration extreme Feedback extreme Communication extreme Unit Testing extreme Feedback 27
extreme Continuous Integration 12 build servers (6 CC) Avg 7-8 builds for each project daily 3000+ JUnit unit, smoke and system tests run every day ~70% good builds even with large refactorings No scheduled drops between milestones builds are used 28
extreme Feedback Firefox CC plugin 29
extreme Feedback (having the desired effect ) 30
extreme Communication Shout for fix Lava Lamp (Red ) Whiteboard for goals Gong for Stand-ups (and lunch ) 31
extreme Communication Weekly stand-up meetings Goals pushed actively On Whiteboard By Email (weekly towards end of cycle) Open Quality Initiative (www.agitar.com/openquality) Reporting of our current dashboards on our public site 32
extreme Unit Testing Eat our own Dog Food 30000+ Agitation Assertions ~ 1 Million+ asserts run every day 25000+ JUnit Assertions run 2-3 times every day 1:1 Test lines:code lines ~ 30% coverage 80%+ coverage with Agitation All Hands Bug-Hunting Test First, Test During, Test Later never Test Never 33
extreme Tracking 34
extreme Tracking 35
The Power of Unit Testing Why Unit Test? The Practical Reality Where do we go from here? What Agitar does Lessons Learned 36
Tests vs. Testing Writing tests is hard 4x test code to 1x code under test Setup code setting up objects in the right state for the interesting outcomes is a majority of test code Tests are a portfolio Unit Tests Handwritten and System Generated Functional and Sequence Tests System Tests Acceptance Tests UI and QA tests The trick is to have lots of tests to characterize and future-proof your system (and then keep them running) 37
The Power of Unit Testing = develop = unit test Horrible Still Bad Oops! Sorry no time to test Oops! Find bugs, but no time to fix them OK Catch few bugs early, time to fix them Good Catch many bugs early, time to fix them Better Hard to fall behind due to bugs Best Time Much harder to fall behind due to bugs Time s Up Check your code in 38
Lessons Learned Large number of tests that don t run frequently are not as useful as small number of tests run continuously Testing needs to be looked as an investment portfolio Mix and Match auto-generated and hand-written Unit Tests, Functional Tests, System Tests Focus on having tests and not just testing And then run them continuously More at www.agitar.com www.developertesting.com www.junitfactory.com www.junit.org 39
Demo 40