Autonomous Real-Time Testing

Transcription

1 Autonomous Real-Time Testing Thomas M. Fehlmann, Zürich Euro Office E: H: 1

2 Dr. Thomas Fehlmann 1981: Dr. Math. ETHZ 1991: for Software Black Belt 1999: Euro Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics 2004: Member of the Board QFD Institute Deutschland QFD Architect 2007: CMMI for Software Level 4 & : Net Promoter Certified Associate 2013: Vice-President ISBSG 2016: Academic Member of the Athens Institute for Education and Research 2

3 Agenda Software Models Software Metrics Autonomous Testing Conclusion 3

4 Agenda Software Models Software Metrics Autonomous Testing Conclusion 4

5 Software Models Modeling Software by ISO/IEC COSMIC Others. Sizing Software according Functionality & Test Intensity Non-functional Characteristics Privacy Safety Other constraints Defined Granularity! 5

6 Modeling User Stories according ISO Data Movements move Data Groups Data Groups categorize what kind of data is moved between objects Data Groups are relevant for security aspects, such as Privacy and Safety Trigger Device User Entry exit Write Functional Process Read Persistent Data exit Entry Application User Data Movements always involve a Functional Process Either Entry & exit, or Read & Write Functional communicate with other Functional either via Devices, or Persistent Data Data Movements moving the same Data Group between the same two objects count as one model elements only Software Boundary 9

7 Modeling User Stories: Know my Way As a Car User I want to let my car take appropriate routes Such that it avoids blocked routes and traffic jams Because I highly value knowing when I ll arrive Data Groups: Destination Location Route with Duration 3 Entry (E) + 1 exit (X) + 0 Read (R) + 1 Write (W) = 5 CFP Navigation Car User Routing Remember Routes Map Service GPS Service Device Car User 1.// Enter Destination Functional Process Persistent Data Other Application GPS Service Routing Remember Routes Map Service GPS Service 5.// Confirm Route 4.// Record Route 3.// Recommend Route 2.// Get Location 11

8 Modeling User Stories: Check my Way As a Car User I want to approve or disapprove the car's choice for routing Such that I can take my preferred route Because I highly value remaining in control Data Groups: Route with Duration Approval-Attribute 3 Entry (E) + 1 exit (X) + 1 Read (R) + 1 Write (W) = 6 CFP Check Car User Routing Remember Routes Approve Route Same Device Car User 1.// Check Route Same Functional Process Same Persistent Data Routing Remember Routes 4.// Approve Route 2.// Get Route 3.// Show Route New Functional Process Last two data movements already implemented Approve Route 5.// Modify Route 8.// Inform 7.// Changed Route 6.// Change Route 12

9 Modeling User Stories: Amend my Way As a Car User I want to optimize my route when needed Such that no incidence blocks my way Because I highly value predicting when I'll arrive Data Groups: Route with Duration Incident/Alert 2 Entry (E) + 0 exit (X) + 1 Read (R) + 0 Write (W) = 3 CFP Same Device Same Functional Process Same Persistent Data Car User Routing Remember Routes Alert Car User Routing Remember Routes Map Service 1.// Routing Alert 2.// Recall Route 3.// Recalculate Route Same Application Last two data movements already implemented Map Service 5.// Inform 4.// Change Route 13

10 Modeling User Stories: Look & Act As a Car User in populated areas I want to let my car reduce speed Such that my car can safely stop, if needed Because I don t like causing delays by an incidence Data Groups: Sensor Data Analysis Results Action 4 Entry (E) + 5 exit (X) + 0 Read (R) + 0 Write (W) = 9 CFP Look Car User Recommender Car Driving Function Neural Network Engine Sensor Bus Sensor App Physical Device Two Functional Three Apps Act 1.// Trigger Sensor 4.// Analysis Request 2.// Start Sensor 3.// Supply Data Sensors Neural Network 6.// Ask for Actions 7.// Recommended Action 5.// Analysis Result Recommender 9.// Inform 8.// Act 14

11 Sample Story Card Story Card for Q001: Populated Area Q001-03Q: Act Story Points: Functional Size: Business Impact: 8 2 y1: 2 Name: Sprint: Paul #02 - Allegretto y4: 3 Test is Ready Draft is Ready Review Done Finalized Approved Functional As a Car User, I want to let my car reduce speed, such that my car can safely stop, so that my car is not causing delays by an incidence Car User Car Driving Function 8.// Act Physical Device 9.// Inform Connect to actuators, and keep the Car User informed On the back of the Story Card are the test cases that concern data movements recorded on the card 15

12 Composing User Stories Join User Stories: Identify common Objects, Data Movements and Data Groups Avoid double work Create an initial Vision of the product Split User Stories: Create Story Cards (in Jira: Tasks) for implementing User Stories over several Sprints Story Cards can address functional & non-functional tasks differently Counting Data Movements yields functional size Story Points complement functional size, indicating expected effort This works because you can easily add, or extract, Data Movement Maps Take care for uniqueness of Data Movements regarding Data Groups This is the essence of the ISO standard 16

13 Look & Act Use sensors for driving Act upon recommendations Navigation Find the best route Alert In case of traffic jams and upcoming obstacles Locate Can I keep with the predicted speed? Check Approve proposed route Look Act Navigation Alert Locate Check The Vision for Autonomous Driving 13 Entry (E) + 12 exit (X) + 3 Read (R) + 2 Write (W) = 30 CFP Car User 10.// Enter Destination 28.// Check Route 31.// Approve Route 32.// Modify Route Recommender 6.// Ask for Actions 7.// Recommended Action 9.// Inform 24.// Update Location 25.// Recalculate Route Car Driving Function 8.// Act 5.// Analysis Result Neural Network Engine 1.// Trigger Sensor 4.// Analysis Request Sensor Bus Sensor App Physical Device Routing Remember Routes Approve Route Map Service GPS Service 2.// Start Sensor 3.// Supply Data 15.// Set Route 16.// Inform 21.// Inform 27.// Inform 12.// Request Route 14.// Record Route 18.// Recall Route 20.// Change Route 23.// Compare with Actual Route 26.// Adapt Route 29.// Get Route 30.// Show Route 33.// Confirm 13.// Recommend Route 17.// Routing Alert 19.// Recalculate Route 11.// Get Location 22.// Update Location 35.// Changed Route 34.// Change Route // Inform

14 Automating Model Creation Traditional Business Case Functional sizing for cost prediction or memory consumption prediction Classical Code Sizing Data Movements correspond to messages between objects Automatic counting exists for Object- Oriented languages and for functional languages such as Scala Many tools available, with various features, mostly but not all from academic sources Widely used in Automotive for programming embedded systems Modern DevOps Business Case Model Creation for Testing Metrics, Privacy & Safety Consumer Metrics Modern Architecture Sizing Data Movements correspond to messages between microservices Network analyzers, and Containerbuilders like Kubernetes provide means that can be used for automatic model creation Currently, to the author s knowledge, no such tools exist out of the box CloudBees offers automated CI/CD Kubernetes support, yielding a model 18

15 Agenda Software Models Software Metrics Autonomous Testing Conclusion 19

16 Agenda Software Models Software Metrics Autonomous Testing Conclusion 20

17 Functional Size Functional Size According ISO 19761, functional size is the number of data movements that move unique data groups between Objects of Interest Functional Persistent Stores Devices, and Other Applications Size is uniquely defined by the granularity of the User Stories that define the functional user requirements Size grows during sprints because granularity refines The rate of this growth is of high interest to agile teams 3 Entry (E) + 1 exit (X) + 0 Read (R) + 0 Write (W) = 4 CFP Functional Process 10.// Move some data 11.// Move some data Other Application 8.// Move some data Functional size does not replace Story Points but complements them Some Device 9.// Move some data Other Device 24

18 Visualizing Software Testing Tester sees selected sequences in the Data Movement Map 3 Entry (E) + 1 exit (X) + 0 Read (R) + 0 Write (W) = 4 CFP Functional Other Some Other Process Application Device Device Tester can walk the data movements when planning or executing tests Makes functionality visible to the development team 10.// Move some data 11.// Move some data 8.// Move some data 9.// Move some data Localizes defects that impact functionality Supports communication between testers, users, and developers 25

19 Measuring Functionality, Test Intensity and Defect Size What happens if data movements don t work as expected, have defects instead? Testers mark and count data movements where defects have been detected One Size Model Test Size = 4 3 Entry (E) + 1 exit (X) + 0 Read (R) + 0 Write (W) = 4 CFP Functional Process 10.// Move some data Other Application 8.// Move some data Some Device Move some data Other Device 11.// Move some data Defect Count = 1 Functional Size: Number of Data Movements needed to implement required functionality Test Size: Number of Data Movements executed in Tests Test Story: Collection of Test Cases aiming at certain User Stories Test Intensity: Total Test Size divided by Total Functional Size Defect Count: Number of Data Movements affected by some defect detected in a Test Story 26

20 Agenda Software Models Software Metrics Autonomous Testing Conclusion 28

21 Agenda Software Models Software Metrics Autonomous Testing Conclusion 29

22 Autonomous Testing requires Artificial Intelligence Testing Automation alone is not good enough Testing Automation without artificial intelligence is likely to fail Too many manual interventions needed that block automation Artificial Intelligence is Classification into sets of similar events, or objects Solutions to the Function y = f x Where y is the known, observable response vector, dimension m x is the unknown, difficult to observe cause vector, dimension n f is a large n m dimensional function mapping cause onto response (effect) Typical methods of solving y = f x are Regression Eigenvector-based Power Methods ( Page-Algorithms ) 30

23 Arrow Terms for Continuous Software Testing x 1, x 2,, x n y Test Data Test Data Test Data Test Result The Powerset of all Arrow Terms is a Combinatory Algebra Powerset means including tests of tests Algebraic combination of tests is a powerful method for enlarging test coverage Test Data refer to specific Data Groups with specific characteristics Data Groups are needed for test automation Test stories are represented by sets of Arrow Terms 31

24 x 1, x 2,, x n y Sample Test Case: Football in Motion detected! Moving, Football Act! Recommender Car Driving Function Neural Network Engine Sensor Bus Sensor App Physical Device 1.// Trigger Sensor 2.// Start Sensor 3.// Supply Data 5.// Analysis Result 4.// Analysis Request 6.// Ask for Actions 7.// Recommended Action 8.// Act 32

25 x 1, x 2,, x n y Sample Test Case: Football in Motion detected! Looking Around, Player Slow! Recommender Car Driving Function Neural Network Engine Sensor Bus Sensor App Physical Device 1.// Trigger Sensor 2.// Start Sensor 3.// Supply Data 5.// Analysis Result 4.// Analysis Request 6.// Ask for Actions 7.// Recommended Action 8.// Act 33

26 x 1, x 2,, x n y Sample Test Case: Privacy of Route Selection Encrypt Decrypt Car User Routing Remember Routes Map Service 18.// Recall Route 17.// Routing Alert 19.// Recalculate Route 21.// Inform 20.// Change Route 34

27 x 1, x 2,, x n y Intelligent Analysis of Test Cases Data Movement 1.// Trigger Sensor appears in User Stories Q001: Populated Area, Q004: Amend my Way, Q006: Able to Stops Data Movement 5.// Finding appears in User Stories Q001 : Populated Area, Q002: Obstacle, Q004: Amend my Way, Q006: Able to Stops Etc. Recommender Car Driving Function Neural Network Engine Sensor Bus Sensor App Physical Device 1.// Trigger Sensor 2.// Start Sensor 3.// Supply Data 5.// Analysis Result 4.// Analysis Request 6.// Ask for Actions 7.// Recommended Action 8.// Act 40

28 1) CT-A.1 2) CT-B.1 3) CT-C.1 4) CT-C.2 5) CT-C.3 6) CT-D.1 7) CT-E.1 8) CT-E.2 9) CT-F.1 10) CT-F.2 11) CT-F.3 Goal Test Coverage People around Obstacle ahead Get route Change route Update position Approval Arrival time Learnings Keep under control Brake action Avoid stops Achieved Coverage Intelligent Analysis of Test Cases Test Coverage Deployment Combinator Test Stories Data Movement Frequency Goal Profile User Stories Q001 Populated Area Q002 Obstacle Q003 Know my Way Q004 Amend my Way Q005 Check my Way Q006 Able to Stop Ideal Profile for Test Stories: Convergence Gap Total Test Size 0.10 Convergence Range 0.20 Convergence Limit Test Coverage Profile Convergence Gap 41

29 How do We Know What is Relevant? We need a Goal Profile Expressing the relative importance of User Stories Teams have such a profile They need it for setting priorities in the Sprints! Compare what s being tested by the Test Stories with the User Story Profile By counting frequency of data movements being executed If the Convergence Gap is close to zero Test Stories test the response that users expect If the Convergence Gap opens Some Test Cases are missing, or superfluous 42

30 Goal Profile for User s Values Privacy Needs (PN) CN PN Safety Needs (SN) CN SN #Privacy Index #Safety Index Autonomous Tests Targets User s Values (UV) US UV How to get a goal profile for User s Values? TS US Privacy Needs Safety Needs User Stories (US) #Functional Size Emotional Needs Business Needs Test Stories (TS) #Defects 43

31 Q001 Q002 Q003 Q004 Q005 Q006 Goal Profile Populated Area Obstacle Know my Way Amend my Way Check my Way Able to Stop Achieved Profile Users Values Transformed into a User Story Profile Users' Values AHP Priorities Users' Values Topics Attributes Weight Profile Y.a Drive Fast y1 Driving Arrive safe Do not block other traffic Have fun 16% 0.36 y2 Smooth Driving Drive predictibly Do not break unnecessarily 15% 0.32 y3 Arrive in Time Arrive predictibly Avoid obstacles 23% 0.50 Y.b Drive Safe y4 Avoid Incidences Drive foresightful Know what's ahead Know my way 27% 0.58 y5 No Surprises Communicate Never surprise anybody Give signs 19% Users' Values Deployment Combinator User Stories Users' Values y1 Driving Count number of Data Movements in User Stories that support particular Users Value y2 Smooth Driving y3 Arrive in Time y4 Avoid Incidences y5 No Surprises Solution Profile for User Stories: Convergence Gap Total Effort Points 0.10 Convergence Range 0.20 Convergence Limit 44

32 Q001 Q002 Q003 Q004 Q005 Q006 1) CT-A.1 2) CT-B.1 3) CT-C.1 4) CT-C.2 5) CT-C.3 6) CT-D.1 7) CT-E.1 8) CT-E.2 9) CT-F.1 10) CT-F.2 11) CT-F.3 Goal Profile Populated Area Obstacle Know my Way Amend my Way Check my Way Able to Stop Achieved Profile Goal Test Coverage People around Obstacle ahead Get route Change route Update position Approval Arrival time Learnings Keep under control Brake action Avoid stops Achieved Coverage Intelligent Selection of Test Cases Test Coverage Deployment Combinator Test Stories Users' Values Deployment Combinator User Stories User Stories Users' Values Q001 Populated Area y1 Driving 0.36 Q002 Obstacle y2 Smooth Driving 0.32 Q003 Know my Way y3 Arrive in Time 0.50 Q004 Amend my Way y4 Avoid Incidences 0.58 Q005 Check my Way y5 No Surprises 0.42 Q006 Able to Stop Total Test Size 0.10 Convergence Range 0.20 Convergence Limit Solution Profile for User Stories: Convergence Gap Ideal Profile for Test Stories: Convergence Gap Total Effort Points 0.10 Convergence Range 0.20 Convergence Limit

33 1) CT-A.1 2) CT-B.1 3) CT-C.1 4) CT-C.2 5) CT-C.3 6) CT-D.1 7) CT-E.1 8) CT-E.2 9) CT-F.1 10) CT-F.2 11) CT-F.3 Goal Test Coverage People around Obstacle ahead Get route Change route Update position Approval Arrival time Learnings Keep under control Brake action Avoid stops Achieved Coverage Now let s turn on the Test Generator Test Coverage Deployment Combinator Test Stories Stay with Test Stories and User Stories User Stories Q001 Populated Area Q002 Obstacle Q003 Know my Way Q004 Amend my Way Q005 Check my Way Q006 Able to Stop Ideal Profile for Test Stories: Convergence Gap Total Test Size 0.10 Convergence Range 0.20 Convergence Limit The Test Generator produces Test Cases that he can prove to yield correct results In terms of privacy protection And in terms of safety in driving Test Status Summary Total CFP: 30 Test Size in CFP: 670 Test Intensity in CFP: 22.3 Defects Found in Total: 0 Defect Density: 0.0% Defects Pending for Removal: 0 Data Moves Covered: 100% 46

34 1) CT-A.1 2) CT-B.1 3) CT-C.1 4) CT-C.2 5) CT-C.3 6) CT-D.1 7) CT-E.1 8) CT-E.2 9) CT-F.1 10) CT-F.2 11) CT-F.3 Goal Test Coverage People around Obstacle ahead Get route Change route Update position Approval Arrival time Learnings Keep under control Brake action Avoid stops Achieved Coverage After a few Test Cases more the Convergence Gap opens Test Coverage Deployment Combinator User Stories Test Stories Q001 Populated Area Q002 Obstacle Q003 Know my Way Q004 Amend my Way Q005 Check my Way Q006 Able to Stop Ideal Profile for Test Stories: Convergence Gap Total Test Size 0.10 Convergence Range 0.20 Convergence Limit Adding Test Cases improves Test Density Here: Test Cases for image recognition in rain, or at night This might lead to lower the focus on Users Values The Convergence Gap opens Test Status Summary Total CFP: 30 Test Size in CFP: 745 Test Intensity in CFP: 24.8 Defects Found in Total: 0 Defect Density: 0.0% Defects Pending for Removal: 0 Data Moves Covered: 100% 47

35 1) CT-A.1 2) CT-B.1 3) CT-C.1 4) CT-C.2 5) CT-C.3 6) CT-D.1 7) CT-E.1 8) CT-E.2 9) CT-F.1 10) CT-F.2 11) CT-F.3 Goal Test Coverage People around Obstacle ahead Get route Change route Update position Approval Arrival time Learnings Keep under control Brake action Avoid stops Achieved Coverage By looking at Missed User Stories we can close it again Test Coverage Deployment Combinator User Stories Test Stories Q001 Populated Area Q002 Obstacle Q003 Know my Way Q004 Amend my Way Q005 Check my Way Q006 Able to Stop Ideal Profile for Test Stories: Convergence Gap Total Test Size 0.10 Convergence Range 0.20 Convergence Limit Using the Convergence Gap as hash, the Test Generator focuses on Test Cases closing it again This is an intelligent selection process Based on Just like Google does And Big Data analysis Test Status Summary Total CFP: 30 Test Size in CFP: 824 Test Intensity in CFP: 27.5 Defects Found in Total: 0 Defect Density: 0.0% Defects Pending for Removal: 0 Data Moves Covered: 100% 48

36 How this Fits into DevOps Test Stories start with User Stories You can add Test Stories any time You should add your Test Stories as soon as you can For Privacy Protection And for Security Run the Test Generator often Daily Build with Daily Tests Always more tests run and get executed You ll find defects more rapidly than anybody else can who relays on traditional methods such as AutoSAR or ISO You ll need quite a bit more Computing Power 49

37 Agenda Software Models Software Metrics Autonomous Testing Conclusion 50

38 Agenda Software Models Software Metrics Autonomous Testing Conclusion Conclusion 51

39 Impact on Safety Privacy Protection Software Metrics for the Automotive Industry 0 Functional Size COSMIC E.g., FP Test Metrics FP Privacy Index & Minimum Privacy For Data Movements Featuring overall Privacy Protection Test Size E.g., FP Privacy Needs 5 Test Intensity E.g., 723`456 23`456 = 30.8 Acceptable Defect Density Security Index & Maximum Risk For Data Movements Featuring overall Safety Risk Exposure E.g., 0 Defects Incurrence Frequency 53

40 Impact on Safety Privacy Protection Software Metrics are Something for the Public 0 We trust software more than humans Human can be distracted, angry, sick, exhibit bad humor 1 2 Software is more predictable, repeatable and is testable Sometimes, we believe software was tested before released However, how should consumers know? We need metrics that consumers can understand For instance, Privacy Protection and Safety Risk exposure Privacy Needs With graphical indicators similar to EC power consumption labels Incurrence Frequency 54

41 Outlook for Developers Autonomous Real-time Testing is an opportunity for DevOps to safely establish agile development methods in safety-critical systems In attending automated tools, the methods presented in this talk are immediately available A book explaining all this stuff is in the making You can set up autonomous testing in agile development Operations can extend testing as the system expands Measuring privacy protection and safety risk exposure is necessary to get acceptance Consumers must be able to see value in privacy and safety testing 55

42 Outlook for Testers Privacy and Safety are domains that can be adequately expressed in formal systems Thus, a theorem prover can be build Automated real-time testing is possible Compliancy to banking or other regulatory systems is also something that can be formalized Automated real-time testing is possible Other testing approaches might be significantly harder If domain cannot be formalized, such as Ease-of-Use or Attractive Design Automated real-time testing works probably not for non-functional quality testing 56

43 Constraints Tools are not ready yet The tools shown here are Excel prototypes not fit for production The mathematics behind transfer functions and combinatory logics are not commonly known What I ve shown here is a glimpse into what s coming upon us with Digitalization However, this future is just around the corner While enthusiasm for autonomous cars has faded, driving assistance systems are abundant and ready to become daily gadgets Every smartphone today needs SW metrics and autonomous real-time testing after downloading new Apps, or interconnecting with new services 57

44 My Tools If you dare to play with the tools that I used for this presentation Here s the link: This is a two-way encrypted read-only tresor, not easily accessible for anyone who tries to modify my VBA code You ll find The COSMIC model Priority settings Test Coverage for Safety & Privacy A full initial Backlog with Story Cards Recommended before trying: Connect with me on LinkedIn Send to thomas.fehlmann@e-p-o.com Or leave me your business card 58

45 Questions? Logos Press Berlin

46 New Book on 60