Evidence-based Development coupling structured argumentation with requirements development Jeremy.Dick@integrate.biz integrate 2012
based on paper Paper: EVIDENCE-BASED DEVELOPMENT COUPLING STRUCTURED ARGUMENTATION WITH REQUIREMENTS DEVELOPMENT IET Safety Edinburgh, September 2012 Experience: application in large UK defence and civil nuclear projects
agenda requirements in brief overview a shift in thinking reviewing traceability traceability, rationale and evidence Evidence-based Development conclusion
requirements in brief manage requirements in layers Subsystem integrate 2014
requirements in brief manage requirements in layers express requirements as traceable statements in appropriate language Subsystem integrate 2014
requirements in brief manage requirements in layers express requirements as traceable statements in appropriate language systematically decompose requirements through the layers and record the relationships Subsystem integrate 2014
manage requirements in layers express requirements as traceable statements in appropriate language systematically decompose requirements through the layers and record the relationships requirements in brief record the rationale for the decomposition compliance information Subsystem integrate 2014
demonstrate compliance requirements in brief Standards and Regulations Standards and Regulations Standards and Regulations Subsystem integrate 2014
plan tests against requirements requirements in brief Standards and Regulations Standards and Regulations Standards and Regulations Subsystem verifies verifies verifies Validation, Verification, Test and Inspection Plan integrate 2014
requirements in brief collect results/evidence against test plans Standards and Regulations Standards and Regulations Standards and Regulations Subsystem verifies verifies Validation, Verification, Test and Inspection Plan EVIDENCE verifies integrate 2014
manage change through impact analysis requirements in brief Standards and Regulations Standards and Regulations Standards and Regulations Subsystem verifies verifies Validation, Verification, Test and Inspection Plan EVIDENCE verifies integrate 2014
assurance case for requirement verifies verifies EVIDENCE verifies integrate 2014
assurance-based development or evidence-based development Standards and Regulations Standards and Regulations Standards and Regulations Subsystem verifies verifies Validation, Verification, Test and Inspection Plan EVIDENCE verifies integrate 2014
a shift in thinking Stakeholder Subsystem Subsystem integrate 2014
a shift in thinking Stakeholder from reviewing requirements in isolation: review one requirements document at a time Subsystem Subsystem integrate 2014
integrate 2014 Stakeholder Subsystem Subsystem a shift in thinking from reviewing requirements in isolation: review one requirements document at a time to reviewing in context: review relationship between requirements at multiple levels because most of the meaning is captured in the relationships, not just in the requirements
traceability documenting how high-level requirements are transformed into low-level requirements understanding how requirements are satisfied, validated and verified explaining compliance against regulations connecting evidence to design integrate 2014 17
tracing: satisfaction relationship User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode.. The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
User Requirement satisfaction argument The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Why do you believe that this traceability is sufficient and necessary?. The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
User Requirement satisfaction argument The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Satisfaction Argument???? Prototyping shows that the user experience is improved by showing all possible states, indicating the current state and indicating the next possible states, so long as there are no more. than 15 possible states. The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
structured argumentation User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Assume that there are fewer than 15 states Satisfaction Argument???? Prototyping shows that the user experience is improved by showing all possible states, indicating the current state and indicating the next possible states, so long as there are no more. than 15 possible states. HCI prototype ref H56a v1 The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
Goal Structuring Notation? User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Assume that there are fewer than 15 states Satisfaction Argument???? Prototyping shows that the user experience is improved by showing all possible states, indicating the current state and indicating the next possible states, so long as there are no more. than 15 possible states. HCI prototype ref H56a v1 The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
claim / argument / evidence User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Assume that there are fewer than 15 states Satisfaction Argument???? Prototyping shows that the user experience is improved by showing all possible states, you indicating believe the current state and indicating the next possible states, so long as there are no more. than 15 possible states. the claim The HCI will present a button for each of the possible operating modes. argument: explains why The HCI will indicate which of the operating modes are reachable by valid transition from the current mode. HCI prototype ref H56a v1 analysis models experiments prototypes The HCI will indicate which is the current operating mode. The HCI will switch operating modes when a button for a new reachable mode is pressed.
tracing: verification relationship User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode.. Acceptance Test Put the facility into a valid operating state, and attempt to select a new valid operating state. Acceptance Test Put the facility into a valid operating state, and attempt to select a new invalid operating state.
verification argument Acceptance Test User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Verification Argument This requirement is verified by considering a positive and a negative attempt at updating software. Put the facility into a valid operating state, and attempt to select a new valid operating state. Acceptance Test Put the facility into a valid operating state, and attempt to select a new invalid operating state.
structured argumentation User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Verification Argument This requirement is verified by considering a positive and a negative attempt at updating software models similarities experience argument: explains why you believe the claim. Acceptance Test Put the facility into a valid operating state, and attempt to select a new valid operating state. Acceptance Test Put the facility into a valid operating state, and attempt to select a new invalid operating state.
structured argumentation User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. Verification Argument This requirement is verified by considering a positive and a negative attempt at updating software Acceptance Test Put the facility into a valid operating state, and attempt to select a new valid operating state. Acceptance Test Put the facility into a valid operating state, and attempt to select a new invalid operating state. argument: explains why you believe the claim Test Argument This test objectives was fulfilled because. test results
Evidence-based Development What is it? framework for collecting evidence for the correctness of a system as you design the system uses requirements traceability as the structure for establishing arguments and supporting evidence extends the structured argument paradigm to cover all kinds of requirement not just safety gives ownership of assurance to every engineer
objectives of EbD connect the assurance case connected to the design the assurance case should not be an after-thought develop the assurance case early in time to influence the design in time to save costly rework late in the day apply a uniform approach to all aspects of assurance address all kinds of requirements: function, performance, ease-of-use, reliability, safety,... have a single point of reference for structured argumentation
relationships in the W model Statement of need Operational use Stakeholder requirements validates Acceptance test plan fulfils Acceptance test results requirements test plan test results Subsystem requirements Integration test plan Integration test results requirements test plan test results
progressive assurance Statement of need Operational use Stakeholder requirements Acceptance test plan Acceptance test results requirements test plan test results Subsystem requirements Integration test plan Integration test results requirements test plan test results
single requirement assurance case
testing mantra test early, test often because the sooner you find defects, the cheaper it is to fix
testing (V&V) covers early: design analysis design modelling design reviews middle: component tests factory tests integration tests late: systems tests operational tests acceptance tests all of these actions collect evidence for the correctness of the design w.r.t. requirements (verification)
V&V as request for evidence User Requirement The operator shall be able to select a new operating mode only if it represents a valid transition from the current operating mode. V&V Methods Evidence This requirement is verified by Analysis Analogy Inspection results results results early supports design intent Satisfaction Argument This requirement is met by... The HCI will present a button for each of the possible operating modes. The HCI will indicate which of the operating modes are reachable... Test Demonstration results results late supports design fulfilment
progressive assurance Statement of need Operational use Stakeholder requirements Acceptance test plan Acceptance test results requirements test plan test results Subsystem requirements Integration test plan Integration test results requirements test plan test results
progressive provision of evidence Statement of need Operational use Stakeholder requirements validates Acceptance test plan Operational product requirements test plan Subsystem requirements Integration test plan Subsystems requirements test plan s Analysis results Analysis results test results Integration test results test results Acceptance test results
the principles of EbD seem sound in practice, a cultural shift is required have to communicate benefits more effectively mentoring in how to write arguments engineers have new emphasis on owning V&V effective tool support is vital in visualising data focussed views of local argument structures navigation of large-scale argument structures going forward we will: start earlier know better how to write arguments place more emphasis on mentoring summary
using structured arguments within requirements development seems sound in practice, a cultural shift is required have to communicate benefits more effectively mentoring in how to write arguments engineers have new emphasis on owning V&V effective tool support is vital in visualising data focussed views of local argument structures navigation of large-scale argument structures next time we will: start earlier know better how to write arguments place even more emphasis on mentoring summary