Ready to Automate? Ready to Automate?

Transcription

1 Bret Pettichord

2 2. Testers aren t trying to use automation to prove their prowess. 3 Monitoring and Logging Diagnostic features can allow you to View history and details of transactions View the state of running methods, threads or processes Attach a debugger and get a stack trace Get reports of unusual behavior Learn about the logging available from web servers, databases and other standard system components 4 2

3 Diagnostic Features These make the software behavior more visible. Assertions. These logical statements in the code make assumptions explicit. If false, there must be a bug. Typically assertion checking is only done during testing and debugging. Event triggers. Provide identifying signals when tasks begin and end. Database Integrity Checks. Ongoing tests for database corruption, making corruption quickly visible to the tester. Code Integrity Checks. Quick check to see whether code has been overwritten. Memory Integrity Checks. Check for wild pointers, other corruption. Access methods. Special access methods allow viewing component state Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 6 3

4 Varieties of Vendorscript C-like languages Look like C, except without pointers Standard idioms aren t supported Visual Basic-like languages Is it really VB (or VBA)? Hybrid languages How will you learn it? How can you tell if it s a language bug? 7 2. Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 4. Testers and developers work cooperatively and with mutual respect. 8 4

5 2. Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 4. Testers and developers work cooperatively and with mutual respect. 5. Automation is developed on an iterative basis. 9 Essential Capabilities Maintainability Will the tests still run after product design changes? Can tests be easily updated for next release? Integrity Can your test results be trusted? Do you get lots of false alarms? Are you sure that failed tests always appear in the test results? Reliabilitity Can you rely on the test suite to actually do some testing when you really need it? Will it run on all the platforms and configurations you need to test? 10 5

6 2. Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 4. Testers and developers work cooperatively and with mutual respect. 5. Automation is developed on an iterative basis. 6. You understand your requirements for automation. 11 Test automation approaches Unit integration testing Test small units in the native programming language API testing Test the product via an application programming interface GUI test automation Use a GUI tool to test via the product user interface 12 6

7 Unit integration testing What is a unit? A class, function or procedure Typically the work product of a single developer Unit isolation testing Test each unit in isolation Create stubs or simulators for units being depended on Expensive and often difficult Unit integration testing Test units in context Allow calls to classes, functions and components that the unit requires Typically cheaper and easier 13 API testing Does your product have an API? Ask Might be undocumented It may be cheaper to create or expose one for testing than to build GUI test automation infrastructure 14 7

8 Test Interfaces Interfaces may be provided specifically for testing. Excel Xconq Any interface is easier to automate than a GUI. Existing interfaces may be able to support significant testing. InstallShield Autocad Interleaf Tivoli Any defined client/server interface 15 Test Interfaces Product Under Test Programming Interface Test Automation Libraries User Interface Presentation Code Programming Interface Domain Code Which interface will be easier to test? 16 8

9 Comparison of Approaches Unit API GUI Who can do it? Tool support required Training required Flexibility Developers Unit test harnesses (public domain) Tests are in same language as product Anyone can run Programmer/ users Scripting test harnesses (public domain) Need to understand API Anyone can run Testers + automation experts GUI test tools (purchase) Requires tool training Must have tool license to run 17 Dimensions of Automation Interfaces Unit, API, GUI Risks Regression, Load, Concurrency, Robustness Test Creation By tester/programmers, by non-programmers, by programs and test models Evaluation Compare with prior results, automated oracles, assertions and flight recorders Coverage Instrumenting the code to understand what s been tested. 18 9

10 2. Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 4. Testers and developers work cooperatively and with mutual respect. 5. Automation is developed on an iterative basis. 6. You understand your requirements for automation. 7. You are open to different concepts of what test automation can mean. 19 Automation Architecture Test Requirements Staff Skills System Under Test Automation Architecture 20 10

11 Opportunities for Automation Platform Setup and Reset Pre-load Database with Testbed Data Smoke Tests Regression Testing Configuration and Multiplatform Testing Load Testing Randomized Testing Code Coverage Measurement Memory Leak and other specialized testing Test compliance with interface standards Collect performance metrics Confirm pre-defined release criteria 21 Staffing Models Domain Experts + Automators Tester/Programmers Junior Programmers Central Automation Team Spare-time Automation Manual Testers with Tools Test Expert + Warm Bodies 22 11

12 2. Testers aren t trying to use automation to prove their prowess. 3. Testability features can be added to the product code. 4. Testers and developers work cooperatively and with mutual respect. 5. Automation is developed on an iterative basis. 6. You understand your requirements for automation. 7. You are open to different concepts of what test automation can mean. 8. Test automation is lead by someone with an understanding of programming and testing. 23 Questions? For more information Software Testing Hotlist Lessons Learned in Software Testing Cem Kaner, James Bach, Bret Pettichord Bret Pettichord bret@pettichord.com