Testing System Qualities

Transcription

1 Testing System Qualities Rebecca Wirfs-Brock Joseph Yoder Copyright 2012 Rebecca Wirfs-Brock, Joseph Yoder, Wirfs-Brock Associates and The Refactory, Inc.

2 Introducing Rebecca President, Wirfs-Brock Associates Agile enthusiast (involved with experience reports since 1 st agile conference, board president Agile Open Northwest) First engineering job in Quality Assurance Pattern enthusiast, author, and Hillside Board Treasurer Old design geek (author of two object design books, inventor of Responsibility-Driven Design, advocate of CRC cards, hot spot cards, & other low-tech design tools, IEEE Software design columnist) Consults and trains top companies on agile architecture, responsiblity-driven design, enterprise app design, agile use cases, design storytelling, pragmatic testing Runs marathons!!!

3 Introducing Joseph Founder and Architect, The Refactory, Inc. Pattern enthusiast, author and Hillside Board President Author of the Big Ball of Mud Pattern Adaptive systems expert (programs adaptive software, consults on adaptive architectures, author of adaptive architecture patterns, metatdata maven, website: adaptiveobjectmodel.com) Agile enthusiast and practitioner Business owner (leads a world class development company) Consults and trains top companies on design, refactoring, pragmatic testing Amateur photographer, motorcycle enthusiast, enjoys dancing samba!!!

4 Some Agile Myths System Qualities can be added at the last moment. We can easily adapt to changing requirements (new requirements). You can change the system fast!!! Don t worry about the architecture.

5 Most testers spend the majority of their time writing functional tests BUT THERE S A LOT MORE TO TEST THAT YOUR SOFTWARE WORKS AS ADVERTISED

6 Pragmatic Testing Issues What kinds of testing should you focus on? Who writes them? Who runs them? When are they run? How are they run?

7 Testing System Qualities Qualities we could consider Usability Security Performance Scalability Internationalization Availability Flexibility Accessibility Location Regulation

8 Discussion What are the important qualities in your system?

9 Functional Tests vs. System Quality Tests Functional: How do I? Tests user stories work as advertised As a reviewer I want to add a note to a chart Compute the charge for an invoice Tests boundary conditions Can I add more than one note at the same place? Are excess charges computed correctly?

10 System Quality Tests How does the system handle? system load? number of add note transactions/minute under normal load system support for? simultaneously updating charts usability? ease of locating and selecting notes Tests that emphasize architecture capabilities and tangible system characteristics

11 Structure for A System Quality Acceptance Test

12 Specify Measurable Results Meter: An appropriate way to measure Scale: The values you expect Natural Scale: time in milliseconds Constructed: 1-10 ranking Proxy: Projecting throughput using sample data

13 Example Performance Scenario Source of Stimulus Users Stimulus Initiate order Artifact System Environment Normal conditions Transactions processed Measure Average latency of 2 seconds Users initiate 1,000 order transactions per minute under normal operations; transactions are processed with an average latency of 2 seconds.

14 Possible Performance Scenario Values Portion of Scenario Source Stimulus Artifact Environment Measure Possible Values External systems, users, components, databases Periodic events, sporadic or random events (or a combination) The system s services, data, or other resources The state the system can be in: normal, overloaded, partial operation, emergency mode Process the event or event sequence and possibly change the level of service Times it takes to process the arriving events (latency or deadline by which event must be processed), the variation in this time, the number of events that can be processed within a particular time interval, or a characterization of events that cannot be processed (missed rate, data loss)

15 Example Security Scenario Source of Stimulus Correctly identified individual Stimulus Tries to transfer money between accounts Artifact Data within the system Environment Normal operation System maintains audit trail Measure Correct data is restored within a day of reported event A known, authorized user transfers money between accounts. The user is later identified as an embezzler by the institution they belong to and the system then restores funds to the original account.

16 Possible Security Scenario Values Portion of Scenario Source Stimulus Artifact Environment Measure Possible Values A human or another system. May be identified (correctly or not) or be unknown. An attack or an attempt to break security by trying to display information, change information, access system services, or reduce system availability The system s services or data. The system might be online or offline, connected to or disconnected from the network, behind a firewall or open to the network. Authenticates user; hides identity of the user; blocks or allows access to data and/or services; records access/modification attempts by identity; stores data in encrypted format; recognizes unexplained high demand and informs a user or other system, or restricts availability. Time/effort/resources required to circumvent security measures with probability of success; probability of detecting attack; probability of identifying individual responsible for attack or access/modification; time/effort/resources to restore data/services; extent to which data/services are damaged and/or legitimate access denied.

17 Example Modifiability Scenario Source of Stimulus Developer Stimulus Artifact Add support UI, Source for new code, service Service code Code Table Environment Modification made with no schema changes Compile time, data definition Measure 2 days to code and test, 1 day to deploy A developer adds support for a new service code to the system by adding the service code to the definitions table and modifying the UI to make it available to users. The modification is made with no data schema changes

18 Possible Modifiability Scenario Values Portion of Scenario Source Stimulus Artifact Environment Measure Possible Values End user, developer, system administrator Wishes to add/delete/modify/vary functionality. Wishes to change some system quality such as availability, responsiveness, increasing capacity. What is to be changed: system user interface, platform, environment, or another system or API with which it interoperates When the change can be made: runtime, compile time, build time, when deployed, Locates places to be modified; makes modification without affecting other functionality; tests and deploys modification End user, developer, system administrator

19 Example Availability Scenario Source of Stimulus Unknown Sensor Stimulus Unexpected Report Artifact System Environment Normal conditions Record raw report in database and log event Measure No lost data An unknown sensor sends a report. The system stores the raw data in the unknown sensor database (to potentially be processed or purged later) and logs the event.

20 Example Usability Scenario Source of Stimulus End user Stimulus Cancel request Artifact System Environment At runtime System backs out pending transaction and releases resources Measure Cancel takes less than one second A user can cancel a pending request within two seconds.

21 Copyright 2012 Wirfs-Brock Associates and The Refactory

22 EXERCISES WRITE A QUALITY SCENARIO FOR A DATA COLLECTION AND ANALYSIS SYSTEM

23 Write A Quality Scenario Using the template handout, write a quality scenario. Be specific. Options: to data received from an unknown sensor. Adding new analyzer plug-ins. Relocating a sensor and being able to correlate data from its previous location (if desired). Detecting and troubleshooting equipment failures Detecting unusual weather conditions and incidents What quality does your scenario address?

24 You can t test warm and fuzzy It should be easy to place an online order TURN VAGUE STATEMENTS INTO CONCRETE MEASURABLE ACTIONS

25 Turning Warm Fuzzies into a Testable Usability Scenario Source of Stimulus Novice user Stimulus Place order Artifact System Environment Web interface with online help Order completed Measure Time to complete order entry task 80% of novice users should be able to place an order in under 3 minutes without assistance. or 80% of novice users should be able to place an order in under 3 minutes only using online help.

26 Agreeing on scenarios and target values

27 Some options Toss out a reasonable number, then discuss to come to a consensus Average informed individuals estimates Use an existing system as baseline Values for similar scenarios Benchmark working code

28 There is more than pass or fail Landing Zone: Lets you define a range of acceptable values Minimal: OK, we can live with that Target: Realistic goal, what we are aiming for Outstanding: This would be great, if everything goes well

29 Landing Zones Roll up product or project success to several key scenarios Easier to make sense of the bigger picture: What happens when one quality scenario edges below minimum? How do others trend? When will targets be achieved? At what cost?

30 Minimum Target Outstanding Performance Throughput (txns per day) 50,000 70,000 90,000 Average txn time 2 seconds 1 second < 1 second Data Quality Intersystem data consistency (per cent critical data attributes consistent) 95% 97% 97% Data accuracy 97% 99% >99% Managing Landing Zones Too many scenarios and you lose track of what s really important Define a core set, organize and group Roll up into aggregate scenarios Re-calibrate values as you implement more functionality be agile

31 Quality Testing Cycle for TDD Identify and Write Quality Scenarios Write Code and Tests Verify Quality Scenarios Clean up Code (Refactor/revise /rework) Ship it!!! Ready to Release? all tests succeed Check all Tests Succeed

32 Test Coverage Can Overlap Integration Tests Smoke Tests Quality Scenarios Acceptance Tests (Functional and qualities) Unit Tests

33 Pragmatic Test Driven Development Is Practical. Testing system qualities can fit into and enhance your current testing. Thoughtful. What qualities need to be tested? Who should write tests? When should you test for qualities? Realistic. You only have so much time and energy. Test essential qualities.

34 Summary Quality Scenarios are easy to write and read. Writing quality test scenarios drives out important cross-cutting concerns. If you don t pay attention to qualities they can be hard to achieve at the last moment. Measuring system qualities can require specialized testing/measurement tools.

35 Resources Agile Myths: agilemyths.com Wirfs-Brock Associates: The Refactory: Joe s website: joeyoder.com Our Pragmatic TDD Course: refactory.com/training/test-driven-development wirfs-brock.com/pragmatictestdrivendevelopment.html Landing Zone blogs: wirfs-brock.com/blog/2011/07/20/introducing-landing-zones/ wirfs-brock.com/blog/2011/08/16/who-defines-or-redefineslanding-zone-criteria/ wirfs-brock.com/blog/2011/08/05/landing-zone-targets-precisionspecificity-and-wiggle-room/ Introducing Pragmatic TDD: wirfs-brock.com/blog/2011/09/23/what-is-pragmatic-testing-all-about/