Software processes, quality, and standards Non-functional testing

Transcription

1 Software processes, quality, and standards Non-functional testing Jaak Tepandi, Jekaterina Tšukrejeva, Stanislav Vassiljev, Pille Haug Tallinn University of Technology Department of Software Science Moodle: Software Quality (Tarkvara kvaliteet) Alternate download: tepandi.ee Version

2 Non-functional testing and analysis - Concept, problems - Types, methods, timing, tools - Testing based on requirements - Method examples Availability - random testing Maturity - fault seeding Accessibility - guidelines, tools Usability, satisfaction - eye tracking, observation, A/B testing

3 Non-functional testing Testing the attributes of a component or system that do not relate to functionality, e.g. reliability, efficiency, usability, maintainability and portability (ISTQB) More difficult to ensure testability of requirements + More difficult to test - How to test in case of long-term requirements? - How to perform very expensive system tests? Types of NF testing: load testing, reliability testing, usability testing, maintainability testing, portability testing, security testing,... Specific NF testing methods - Testable requirements often indicate how to test - A/B testing - Seeded errors, random testing, penetration testing,... In all stages of software development - More often on validation or system testing level Different tools for testing different NF requirements

4 The system requirements specification shall describe: Types Functions and capabilities of the system Business, organizational and user requirements Safety, security, human-factors engineering (ergonomics), interface, operations, and maintenance requirements Design constraints and qualification requirements See also product quality and quality in use (quality=>requirements) (ISO/IEC , )

5 Product quality (ISO/IEC 25010) Quality characteristic => Example requirement => How to test? => Case study? ISO/IEC Functional suitability Functional completeness Functional correctness Functional appropriateness Performance efficiency Time behaviour Resource utilization Capacity Compatibility Co-existence Interoperability Usability Appropriateness recognizability Learnability Operability User error protection User interface aesthetics Accessibility Reliability Maturity Availability Fault tolerance Recoverability Security Confidentiality Integrity Non-repudiation Accountability Authenticity Jaak Tepandi Maintainability Modularity Reusability Analysability Modifiability Testability Portability Adaptability Installability Replaceability

6 Tool examples When? When not? Costar Syst. requirem. analysis Softw. requirem. analysis Testing levels Tests Changes Non-functional Selenium JMeter Load Security User Canoo Random Fuzz System testing Validation testing Risk based Expert Monkey Black box Static Fault seeding Usability tests Smoke Data driven Mock Software design Integr./testing 3 10 Verification White box xunit Softw. programming Module testing Error here = 1 Method examples

7 Reliability ISO/IEC reliability - degree to which a system, product or component performs specified functions under specified conditions for a specified period of time maturity - degree to which a system, product or component meets needs for reliability under normal operation availability - degree to which a system, product or component is operational and accessible when required for use fault tolerance - degree to which a system, product or component operates as intended despite the presence of hardware or software faults recoverability - degree to which, in the event of an interruption or a failure, a product or system can recover the data directly affected and re-establish the desired state of the system

8 NF testing by requirements: SLAs Requirements often in SLAs How to test a testable requirement, e.g. "In case of the environment XYZ, usage profile UVW, activity profile NML, the system may be unavailable for maximum one hour monthly"? What about a requirement involving ten years? A flight control system? An accident? Random testing, static methods, simulations, large scale testing, monitoring, learning,...

9 Availability requirements and SLAs Service - means of delivering value for the customer by facilitating results the customer wants to achieve Service level agreement (SLA) - documented agreement between the service provider and customer that identifies services and service targets Delivering service may require software, processes, people, infrastructure, etc SLAs - customer-level requirements => software, process, etc requirements Examples

10 SLA content: how to test? (examples, ISO/IEC :2012) Contract related items (eg, SLA change control) Service quality related items, eg: service hours, date exceptions, critical business periods and out-of-hours coverage scheduled and agreed interruptions to services, including notice to be given customer responsibilities, e.g. correct use of systems, adherence to the information security policy service provider liability and obligations, e.g. security escalation and notification process complaints procedure service targets upper and lower workload limits, e.g. the ability of the service to support the agreed number of users/volume of work, system throughput actions to be taken in the event of a service interruption SLAs: only most important aspects of the service for the business and the customer

11 Random testing From the very beginning of software programming? For testing reliability, performance, etc Efficient if: an operational profile is evaluated test cases are selected based on this profile a pseudo-random generation algorithm is used for test generation additionally boundary values are tested Important additional results (based on statistical analysis): reliability estimation estimation of remaining errors Problems: profile not always available; evaluating test outputs may be laborious; boundary values may remain untested Option: a parallel implementation for test results

12 Random testing Random inputs User profile Errors Reliability estimation Estimation of remaining errors

13 Example Errors: week 35-20, 36-10, 37-6, 38-3, 39-4, 40-2 Failures: week 35-43, 36-25, 37-15, 38-8, 39-8, 40-5 Reliability? No of remaining errors?

14 Maturity evaluation: Fault seeding The process of intentionally adding known defects to those already in the component or system for the purpose of monitoring the rate of detection and removal, and estimating the number of remaining defects. [IEEE 610] fault seeding tool: A tool for seeding (i.e. intentionally inserting) faults in a component or system. Can also be used to evaluate testing

15 Example Seeded 30 X faults Found 50, incl 10 seeded Found faults Estimation of No of errors X=(50-10)*(30/10-1)

16 concept preconditions advantages disadvantages Summary about the method results relationship to other methods evaluation tools

17 ISO/IEC Usability usability - degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use appropriateness recognizability - degree to which users can recognize whether a product or system is appropriate for their needs learnability - degree to which a product or system can be used by specified users to achieve specified goals of learning to use the product or system with effectiveness, efficiency, freedom from risk and satisfaction in a specified context of use operability - degree to which a product or system has attributes that make it easy to operate and control user error protection - degree to which a system protects users against making errors user interface aesthetics - degree to which a user interface enables pleasing and satisfying interaction for the user accessibility - degree to which a product or system can be used by people with the widest range of characteristics and capabilities to achieve a specified goal in a specified context of use

18 People with special needs in Estonia (10%) 15% in the world (WHO) % in USA 2,550,000 blind people in Europe 23,800,000 low vision 26,350,000 blind in total

19 Accessibility requirements examples WCAG In USA: ion_act_of_1973, -it/about-the-section-508-standards In Estonia: Testing:

20 Examples: Web Content Accessibility Guidelines (WCAG) Non-text Content: All non-text content that is presented to the user has a text alternative that serves the equivalent purpose, except for the situations listed below. (Level A) Use of Color: Color is not used as the only visual means of conveying information, indicating an action, prompting a response, or distinguishing a visual element. (Level A) Resize text: Except for captions and images of text, text can be resized without assistive technology up to 200 percent without loss of content or functionality Keyboard: All functionality of the content is operable through a keyboard interface without requiring specific timings for individual keystrokes, except where the underlying function requires input that depends on the path of the user's movement and not just the endpoints. (Level A)

21 Quality model for quality in use Effectiveness Efficiency Satisfaction Usefulness Trust Pleasure Comfort Freedom from risk Economic risk mitigation Health and safety risk mitigation Environmental risk mitigation Context coverage Context completeness Flexibility Jaak Tepandi

22 Quality in use 1 Quality in use effectiveness - accuracy and completeness with which users achieve specified goals efficiency - resources expended in relation to the accuracy and completeness with which users achieve goals satisfaction - degree to which user needs are satisfied when a product or system is used in a specified context of use usefulness - degree to which a user is satisfied with their perceived achievement of pragmatic goals, including the results of use and the consequences of use trust - degree to which a user or other stakeholder has confidence that a product or system will behave as intended pleasure - degree to which a user obtains pleasure from fulfilling their personal needs comfort - degree to which the user is satisfied with physical comfort

23 Usability testing 1 In a usability test, representative users often try to do typical tasks with the product, while observers, including the development staff, watch, listen, and take notes Usability test typical goals: identify any usability problems that the product has collect quantitative data on participants' performance determine participants' satisfaction with the product

24 Usability testing 2 Typical questions How to select tests? How large group? How often to test? Methods Evaluate time spent on given tasks Eye movement tracking User comments on activities performed Questionnaires and statistical analysis

25 Usability testing 3 Usability testing is easier in case of testable requirements, example: Given the XXX system in YYY environment; 6 motivated users with BBB bacground; TTT tasks. At least 5 of the 6 users must solve the tasks within 20 min. Principles: Testing the site not the users Rely more on what you learn about performance than preference Make use of what you learn Try to find the best solution given the reality of your many users

26 A/B testing Segmenting the target group (two or more subgroups) - subgroups representative of the total population Creating two or more versions of software with one varying element that might affect user behaviour Testing subjects' response to different versions Analysing if the differences are statistically significant Applying the version which gave significantly better results Many arrangements. A form of statistical hypothesis testing

27 Product quality (ISO/IEC 25010) Quality characteristic => Example requirement => How to test? => Case study? ISO/IEC Functional suitability Functional completeness Functional correctness Functional appropriateness Performance efficiency Time behaviour Resource utilization Capacity Compatibility Co-existence Interoperability Usability Appropriateness recognizability Learnability Operability User error protection User interface aesthetics Accessibility Reliability Maturity Availability Fault tolerance Recoverability Security Confidentiality Integrity Non-repudiation Accountability Authenticity Jaak Tepandi Maintainability Modularity Reusability Analysability Modifiability Testability Portability Adaptability Installability Replaceability

28 Takeaway /learning outcomes Non-functional testing Availability. Maturity. Accessibility. Usability... Testing methods Tests defined by requirements Random testing. Fault seeding. Accessibility analysis. A/B testing... Summary about the methods: concept, preconditions, advantages, disadvantages, results, relationship to other methods, evaluation, tools

29 Additional reading (examples) Web Content Accessibility Guidelines, Ian Sommerville. Software Engineering. Ninth Edition. Addison-Wesley, Ch 8. Daniel Galin, Software Quality assurance from theory to implementation, Pearson - Addison-Wesley. Chapter 9, 10. Guide to the Software Engineering Body of Knowledge (SWEBOK), IEEE. Chapter 4. Certified Tester Foundation Level Syllabus, ISTQB. Moodle: Software Quality (Tarkvara kvaliteet). Alternate download: tepandi.ee