SE420 Software Quality Assurance

Transcription

1 SE420 Software Quality Assurance Lecture 11 Testing Automation and Process Improvement October 28, 2014 Sam Siewert

2 Assignment #5 Due 11/9 Reminders Remaining Assignments [Top Down / Bottom-Up] #6 Complete Code, Refine and Run all V&V Tests and Deliver Track Bugs with Bugzilla - Update your Project Code on GitHub - Assignment #6 Posted this Wed, Due 12/1, Late 12/4 Sam Siewert 2

3 Regression Testing and Test Automation Re-test Units After Defects are Fixed Re-run I&T and System Tests Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results Regression Testing (Test Automation) Sam Siewert 3

4 Common Languages Used for SQA Historically Automation BASH and CSH Shell Scripting Still Used Today, but for Quick, Simple, Test Automation [What We ll Cover Today] TCL/TK - Tool Command Language, Expect Larry Wall s Perl Language In Addition and/or More Often Today Python and Numerous Libraries such as robotframework MS PowerShell as well as BASH DejaGnu Ruby using cucumber We will Learn to Automate with BASH (Option to Automate V&V with DejaGnu) Sam Siewert 4

5 Test Case, Test Set Automation Compose Drivers, C main(char * argv[], int argc) to run Test Cases based on Argument Vector Each Driver is a Major Test Case with Numerous Sub-test Cases Theme for Each, such as Performance Data Compare and Verification Error Handling with Fault Injection Run Each Driver from C-shell, Perl, or Python Script to Automate Execution of Test Cases [Vectors] in Test Set Save Results to a Log File Process Log File to Generate [PASS/FAIL] Report Sam Siewert 5

6 Simple Automation Example Test/blob/master/raid_regression.sh Automated Test Run for RAID Always Do git pull and Build First Sam Siewert 6

7 RAID5 Encode Testing Rebuild Erased Sector from XOR Parity Dump Both Data Compare Visually Verify or Automate with Expect / DejaGNU Also Dumps Binary Files for diff compare TEST SET 2 Sam Siewert 7

8 TCL / Expect / DejaGNU Expect (NIST Tool Don Libes) built on TCL - Extension of TCL and Expect, To Automate Build for Test Concept Automate Verification of Expected Results and Convert Test Output o PASS / FAIL Useful for Regression Testing Test Sets or Suites Composed of Many Test Cases DejaGnu Manual Howto: Using DejaGnu for Testing Sam Siewert 8

9 TEST SET 2 Test 1000 Unique (or more) Cases of Rebuild with Assert to Catch Data miscompare External Binary File Diff Expected differences Sam Siewert 9

10 TEST SET 3 Performance Adapted from Earlier Driver Improve to Add Cases for Encode Case for Rebuild of N Sectors Read-Modify Write Encode Cases Fill-out Test Set with More Cases and Verify with DejaGnu Sam Siewert 10

11 Expect PASS/FAIL Automation List all Expect patterns for FAIL and timeout and exit if FAIL matches On timeout expiration, note and go on to next test On PASS match, note and go on Expected PASS Input to Expect Script No Input to Expect Script Sam Siewert 11

12 Test Automation and SEI CMM Coverage Criteria, Bug Open/Close Rate Metrics, Pass/Fail Statistics Provide Quantitative Management [Level 4] Automation is a Step Toward Optimization Necessary, but Not Sufficient (Metrics for Feedback Process Improvement) SEI CMM Levels Feedback Used to Improve Process with Automation and Innovation Entry/Exit Criteria for Phases, Metrics Drive Decisions in Process [Gates] E.g. Boehm s V and Spiral Model, Applied to All Projects - Repeatable Clear Phases with Key Deliverables Analysis, Design, Units, Test Plans Ad Hoc, Code & Fix Driven by Individual Styles, Un-coordinated, More of an Art than Practice Sam Siewert 12

13 SEI CMM Levels Described Level 3 and Above Requires IPPD Customers Managers Engineers Users SQA & Audit To Contribute to Improvement of Process over Lifecycle and Products Sam Siewert 13

14 SEI CMM 4 & 5 Most On-Going Software and Systems Organizations are SEI CMM 3 to Survive [Required for Business Continuity] Levels 4 & 5 Speed Time to Market, Improve Customer Experience, Grow Market Share, Reduces Lifecycle Cost Sam Siewert 14

15 Discussion and Q&A System Testing Entrance Criteria [When is I&T Complete?] Of All Recognized System Test Types, Which are Critical to My Organization? Wikipedia on System Test Types Select Focus in System Testing Based on System Requirements and Product Exit Criteria Regression Test Automation Checkout, Build, Test Set (Cases in Drivers) Scripted, Expected Results Verified & Validated Unit, I&T, System Test Sets Re-run Daily, Weekly, Prior to Push SQA Process Documentation, Audits, Improvement Certification E.g. ISO 9000, SEI CMM Levels 3 5, Etc. Is Process Documented? Repeatable? Can it Be Improved? Automated? Verifiable Criteria? [E.g. Code Coverage Tool Records] Does Certification Matter? Who Does Audits? Current Issue How are Auditors Accountable? Sam Siewert 15