Requirements Testing: Turning Compliance into Commercial Advantage. Mike Bartley, Test and Verification Solutions

Size: px

Start display at page:

Download "Requirements Testing: Turning Compliance into Commercial Advantage. Mike Bartley, Test and Verification Solutions"

Lucas Alexander
5 years ago
Views:

1 Requirements Testing: Turning Compliance into Commercial Advantage Mike Bartley, Test and Verification Solutions 1

2 Agenda Business advantages Some theory Requirements management Mapping requirements to tests Some reflection Some practice Using SQL Recording test results 2

Specifying requirements and features Buying a laptop? What are my requirements? Speed? Connectivity? Weight? Storage? Screen size? Intel Core i52430m processor 2.40 GHz with Turbo Boost up to 3.

3 Specifying requirements and features Buying a laptop? What are my requirements? Speed? Connectivity? Weight? Storage? Screen size? Intel Core i52430m processor 2.40 GHz with Turbo Boost up to 3.00 GHz 15.6" HD WLED TrueLife (1366x768) with 2.0 Mega Pixel Integrated Camera 2 S DVD : XPS L502x Diagnostics and Drivers 2 S 4GB (2X2GB) 1333MHz DDR3 Dual Channel 2 S 500GB Serial ATA (7200RPM) 3

4 So what about software requirements? Software requirements are much more complex missed deadlines exceeded budget inability to meet project reqs Poor and Changing Requirements have been the main cause of project failures for years Bull 1998 (203 interviews) major causes of failure: Highest = breakdown in communications (57%) Chaos Report 1995 (308 managers): Highest = incomplete requirements (13.1%) Agile attempts to solve this issue with requirements 4

5 Capturing software requirements Over the years we have learned many ways to capture requirements: Documents (User / Marketing Requirements Documents) Use Cases or Stories Specification by Examples Tests as Specifications Formal specifications But how do we make sure Requirements are implemented And tested Some industries Mandate this 5

6 Ensuring Requirements are Implemented and Tested Product A Req B Req C Req D Requirements Feature E Feature G Feature F Features Impl H Impl I Implementation Test 1 Test 2 Test 3 Testing 6

7 Example Requirement Client Service Operator finds all current clients in the system where their total value of sales between 2 specified dates is above a specified value Features User (with suitable privilege) can search system for current clients where total value of sales between 2 specified dates is above a specified value User Interface allows a user to select the report, select date range etc Implementation Aspects For every transaction in the system the following data must be stored Revenue value, Date and Client reference Need a query to find clients where total transaction value between 2 dates is above specified value Authorisation, Exceptions, Etc. Tests Query test database XXX with dates d1/m1/y1 and d2/m2/y2. Expected result = client1,... 7

8 Ensuring Requirements are Implemented and Tested Product A Client Service Operator finds Requirements User Interface Search Features Query Implementation Test 1 Test 2 Test 3 Test 4 Query test database XXX Testing 8

9 Beyond tests! Product A Req B Req C Req D Requirements Feature E Feature G Feature F Features Impl H Function Impl I FSM Implementation Function Coverage Test 1 FSM Coverage Line Coverage Assertion Test 2 Test 3 Signoff 9

10 Test Holes and Test Orphans! Product A Uni Directional Req B Req C Req D Req X Feature E Feature F Feature Y Feature G Impl H Impl I Impl Z Test 2 Test 3 Test 4 10

11 Test Orphans and Test Holes Test Orphans Waste Time and Effort! How many tests do you have where you cannot remember what they are testing? What % of your test suite is like this? How much time do you spend running tests where you are not sure of the value of the test? Test Holes Introduce Risk! A requirement is missing a test Do you know how many of your requirements are not tested? 11

12 Mapping requirements to tests Many use a traceability matrix Which supports requirements tracing But will not support status, results and history Requirement Identifiers Reqs Tested REQ 1.1 REQ 1.2 REQ 1.3 Test Cases x x x x x x x x 12

13 Requirements management has good tool support Example tools: Doors, Reqtify, Enterprise Architect, Jira, Requirements get mapped down to features, design, units and can even get to code Until it comes to testing At best they just map to tests without any connection to Test status Test results Results history Making it difficult to track progress 13

14 Requirements tracing the ability to follow the life of a requirement, in both a backward and forward direction [Gotel and Finkelstein, 2006] Requires bidirectional relationships in the requirements tree Advantages Orphan features/code Business advantages later! 14

15 Recap Requirements management helps to record requirements and manage their implementation Can identify test holes Bidirectional requirements mapping allows us to trace in both directions Identify orphan code and tests And we will see impact and risk analysis Want to also associate test status to requirements 15

16 Using SQL for Requirements Management and Testing Reqs Regression Scripts Results Regression Tests Reqs Signoff Test Holes Reqs History Test History 1. Versions 2. Pass/Fail Reqs Configuration System Resources 1. Staff 2. Hardware SQL Database 16

17 Experiences and Recommendations Importing requirements into the SQL database Using XML Able to export back using XML Add a simple API to get test results into the database Regression started, configuration information Test started, test status Regression complete Extract coverage information automatically And store automatically into the database 17

18 Experiences of applying RequirementsBased Testing Supporting Sequential Development Reqs Tracing Tool Reqs Defn Capture specs and mapping Acceptance Test Spec Automate capture of results Acceptance Test Parallel development of verif plans and verification (TTM) Feature Spec Unit Spec Feature Test Spec Integration Test Spec Unit Test Spec Prod Test Integration Test Unit Test Improve specifications through test definitions Map tests to reqs, features, etc, Unit Build Static Analysis Capture results 18

19 Experiences of applying RequirementsBased Testing Supporting Sequential Development Important in building rigorous software systems U.S. Food and Drug Administration (FDA) Mandatory for CMMI level 2 and above, Mandatory for Certification in Aeronautics (DO178B, DO254), Railway Transportation (EN5012x), Automotive (ISO26262,), Medical Systems (FDA 21 CFR), Other (IEC 61508),. 19

. Agree Feature Spec Define tests Map tests to features Not strictly scrum?

20 Experiences of applying RequirementsBased Testing Supporting Iterative Development Product Backlog Sprint Sprint Sprint Sprint 4, 5,.. Agree Feature Spec Define tests Map tests to features Not strictly scrum? Beta release Execute tests Record results in DB Production release Execute tests Record results in DB Maintain feature Execute tests Record results in DB 20

21 So what do we want to track regarding testing? Start tracking testing from the start Are tests defined, written, executing, passing? 21

22 Recap Requirements management helps to record requirements and manage their implementation Bidirectional requirements mapping allows us to trace in both directions Identify orphan code and tests Record multiple test status rather than just pass/fail defined, written, executing, passing Using an SQL database to record test data The mapping The status We will now see how this can create significant business advantage 22

23 Riskbased Testing Product A Risk B Risk B.1 Test 1 Test 2 Test 3 Risk Search for clients whose total value of sales between 2 specified dates is above a specified value returns wrong result Probability = Low Impact = High Tests Risk Query test database XXX with dates d1/m1/y1 and d2/m2/y2. Expected result = client1,... Searching the test database Prioritise tests according to risk Calculate remaining risk A passing test mitigates risk 23

24 Improved TimeToMarket through Prioritisation and Risk Analysis Requirement Priority Unit 1 Integration System Acceptance Req 1 1 Req 2 1 Req 3 1 Req 4 1 Req 5 1 Req 6 2 Req 7 2 Req 8 3!! Req 9 1 Release Unit level with known risk Close at higher level Release with known risk 24

25 Filtering Requirements based on Customers Customer 1 Product A Req B Req C Req D Customer 2 Feature E Feature G Feature F Can we release to customer X? Impl H Impl I Test 1 Test 2 Test 3 25

26 Filtering Based on release Product A Req B Req C Req D Feature E Feature F Feature G Impl H Impl I Test 1 Test 2 Test 3 Beta Release Prodn Release 26

27 Impact Assessment How often do people ignore testing when assessing the impact of a change? What is the impact on changing Req C? Req B Product A Req C Req D Feature E Feature F Feature G Impl H Impl I Test 1 Test 2 Test 3 27

28 Impact Assessment What is the impact on dropping test 3? Req B Product A Req C Req D Feature E Feature F Feature G Impl H Impl I Test 1 Test 2 Test 3 28

29 Return on Investment? Initial investment is relatively high Building initial SQL database Ensuring requirements are recorded Ensuring test information is stored Mapping requirements to tests Business advantage is huge Identify test holes and test orphans Understanding status at all points in project Defined, written, running, passing Automation of analysis (risks, impact, release readiness) Trend analysis Better prediction of release readiness 29

30 Summary Map requirements to tests Database to record mappings and results Store results from test automation Record % s of tests defined, written, executing, passing Advantages Identify test holes and test orphans Track the status of the whole verification effort Use historical perspective for more accurate predictions Better reporting of requirements status Support for Riskbased testing Prioritisation and Risk Analysis Filtering Requirements based on Customers and releases Impact analysis Support for regulatorybased requirements signoff 30

31 Q&A Mike Bartley Mike Bartley on LinkedIn Other materials on 31

(Complete Package) We are ready to serve Latest Testing Trends, Are you ready to learn? New Batches Info

(Complete Package) We are ready to serve Latest Testing Trends, Are you ready to learn? New Batches Info (Complete Package) WEB APP TESTING DB TESTING We are ready to serve Latest Testing Trends, Are you ready to learn? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME