SE420 Software Quality Assurance

Transcription

1 SE420 Software Quality Assurance Lecture 11 Using Test Automation for Process Improvement [SEI Level 3,4,5 Defined, Repeatable, Measureable, Continuously Improved] March 26, 2018 Sam Siewert

2 Reminders Assignment #5 Remaining Assignments [Top Down / Bottom-Up] #6 Complete Code, Refine and Run all V&V Tests and Deliver Track Bugs with Bugzilla - or GitHub Issues or Spreadsheet (backup) Update your Project Code on GitHub - Assignment #5 & #6 Posted Questions? Sam Siewert 2

3 Regression Testing and Test Automation Re-test Units After Defects are Fixed Re-run I&T and System Tests Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results Regression Testing (Test Automation) Sam Siewert 3

4 Common Languages Used for SQA Automation Historically BASH and CSH Shell Scripting Used for quick and simple test automation TCL/TK - Tool Command Language, Expect Larry Wall s Perl language (found in older test suites) In Addition and/or More Often Today, [PL specific] Python and numerous libraries such as robotframework MS PowerShell, Autoit for Windows and BASH for Linux DejaGnu for most any compiled PL Java and JUnit, JCov Ruby using cucumber Team #2 - JavaScript - Jasmine, AVA, Mocha Examples to Automate C/C++ with BASH (Expect) - Example SRS gcov, gprof METRICS assert (why?) TEST CONTROL Expect TEST AUTOMATION Makefile, BASH TEST SCRIPTING Sam Siewert 4

5 Test Case, Test Set Automation Compose Drivers, C main(char * argv[], int argc) to run Test Cases based on Argument Vector Each Driver is a Major Test Case with Numerous Sub-test Cases Theme for Each, such as Performance Data Compare and Verification Error Handling with Fault Injection Run Each Driver from C-shell, Perl, or Python Script to Automate Execution of Test Cases [Vectors] in Test Set Save Results to a Log File Process Log File to Generate [PASS/FAIL] Report Sam Siewert 5

6 Simple Automation Example Test/blob/master/raid_regression.sh Automated Test Run for RAID Always Do git pull and Build First Sam Siewert 6

7 RAID5 Encode Testing Rebuild Erased Sector from XOR Parity Dump Both Data Compare Visually Verify or Automate with Expect / DejaGNU Also Dumps Binary Files for diff compare TEST SET 2 Sam Siewert 7

8 TCL / Expect / DejaGNU Expect (NIST Tool Don Libes) built on TCL - Extension of TCL and Expect, To Automate Build for Test Concept Automate Verification of Expected Results and Convert Test Output o PASS / FAIL Useful for Regression Testing Test Sets or Suites Composed of Many Test Cases DejaGnu Manual Howto: Using DejaGnu for Testing Sam Siewert 8

9 TEST SET 2 Test 1000 Unique (or more) Cases of Rebuild with Assert to Catch Data miscompare External Binary File Diff Expected differences Sam Siewert 9

10 TEST SET 3 Performance Adapted from Earlier Driver Improve to Add Cases for Encode Case for Rebuild of N Sectors Read-Modify Write Encode Cases Fill-out Test Set with More Cases and Verify with DejaGnu Sam Siewert 10

11 Expect PASS/FAIL Automation List all Expect patterns for FAIL and timeout and exit if FAIL matches On timeout expiration, note and go on to next test On PASS match, note and go on Expected PASS Input to Expect Script No Input to Expect Script Sam Siewert 11

12 General V&V Automation Many Tools Exist DejaGnu a short introduction Go Beyond Batch Test and Visual Inspection Batch is Execution Automation Only To Run Drivers in A Sequence Expect / DejaGnu Automates V&V Reports in Addition to Running Drivers Sam Siewert 12

13 Test Automation and SEI CMM Coverage Criteria, Bug Open/Close Rate Metrics, Pass/Fail Statistics Provide Quantitative Management [Level 4] Automation is a Step Toward Optimization Necessary, but Not Sufficient (Metrics for Feedback Process Improvement) SEI CMM Levels Feedback Used to Improve Process with Automation and Innovation Entry/Exit Criteria for Phases, Metrics Drive Decisions in Process [Gates] E.g. Boehm s V and Spiral Model, Applied to All Projects - Repeatable Clear Phases with Key Deliverables Analysis, Design, Units, Test Plans Ad Hoc, Code & Fix Driven by Individual Styles, Un-coordinated, More of an Art than Practice Sam Siewert 13

14 SEI CMM Levels Described Level 3 and Above Requires IPPD Customers Managers Engineers Users SQA & Audit To Contribute to Improvement of Process over Lifecycle and Products Sam Siewert 14

15 SEI CMM 4 & 5 Most On-Going Software and Systems Organizations are SEI CMM 3 to Survive [Required for Business Continuity] Levels 4 & 5 Speed Time to Market, Improve Customer Experience, Grow Market Share, Reduces Lifecycle Cost Sam Siewert 15

16 SEI Rating Examples Certification for Organizations and Professionals cation/ ing/certificates/holders urement/ (methods for Level- 4, 5) Organizations Claiming SEI CMM Level 4 or 5 More Today Based on Automation, Life-long Learning Sam Siewert 16

17 Note on Validation and Verification Metrics, Repeatability and Optimization Layer on Good Base of V&V Did We build the right thing Validation Did We build the thing right Verification For tests, did we build the test right? Test verification Test your tests Follow good practices in Test driver code Did we build the right test? Test validation Are we running the test on correct Architecture and HW configuration? Are we running the test with the correct and most current build of the software? Sam Siewert 17

18 Discussion and Q&A Does SEI CMM Rating Matter? System Testing Entrance Criteria [When is I&T Complete?] Of All Recognized System Test Types, Which are Critical to My Organization? Wikipedia on System Test Types Select Focus in System Testing Based on System Requirements and Product Exit Criteria Regression Test Automation Checkout, Build, Test Set (Cases in Drivers) Scripted, Expected Results Verified & Validated Unit, I&T, System Test Sets Re-run Daily, Weekly, Prior to Push SQA Process Documentation, Audits, Improvement Certification E.g. ISO 9000, SEI CMM Levels 3 5, Etc. Is Process Documented? Repeatable? Can it Be Improved? Automated? Verifiable Criteria? [E.g. Code Coverage Tool Records] Does Certification Matter? Who Does Audits? Current Issue How are Auditors Accountable? Sam Siewert 18