Requirements Spec. Design Coding and Unit Testing Characteristics of System to be built must match required characteristics (high level) Architecture consistent views Software Engineering Computer Science 520-620 Spring 2014 Prof. Leon Osterweil Test Results must match required behavior Hi level design must show HOW requirements can be met Test plan exercises this code (low level) specification Code must implement design Code Test Plan The Coding Phase Coding vs. Code Goal: Produce executable code in a coding language Gets down to very specific details: --Procedures/algorithms --Data structures --The interactions between them --THE DEVIL IS IN THE DETAILS Coding is usually 10-15% of the effort on a software development project: We will spend little time on it in this course Coding should follow closely the specifications resulting from the final phases of design --Modular structure of the code --Object specifications (data modules) (Data abstractions) Here too more is know about the artifact than the activity Coding How should we create code? Helps to have a detailed low-level design Helps to have some patterns and idioms But iteration/rework/refactoring will be needed When, and how do we know when to do what? Code Choice of notations here too» different coding languages But same criterion» What is best at meeting stakeholder needs? Coding Goal: Create code that can be executed on a computer Developer writes source code Object code emitted from a compiler So, is source code really just another model? Executable results from loading object code with libraries, utilities, etc. Important to keep all of these straight Some languages are designed to support specific design methodologies Some languages are special-purpose, well adapted to specific application domains Overall Coding Language Trends Less focus on being close to the machine More focus on integrating with development lifecycles Different language approaches for different lifecycle approaches Static languages for critical software» Emphasis on modularity, testing support, etc. Dynamic, high-level languages for applications in highly competitive situations» Dynamic, late-binding, powerful semantic features Increasing concern for security
What makes a coding language good? What makes a coding language good? If it meets the needs of its stakeholders A good language is one that meets the needs of its stakeholders Different kinds of projects Quality is super-important Rapid deployment is key Evolvability is paramount Emphasis on user interface Etc. Suggest languages with strengths like Readability Expressive power Low level (close to the machine) Dynamism and late binding On Languages Bad code can be written in any language But some languages encourage bad practices Good code can be written in any language But some languages encourage it/make it easier And discourage bad practices Most modern languages try to encourage good practices Like those we have been advocating (in discussing design)» Modularity» Information hiding» Data abstraction» Incorporation of design and requirements specification into code» Support for testing and analysis» Concern for security CS 620 Spring 2014 Etc. Univ. of Massachusetts Information Hiding in Implementation Implementation units should hide internal details as specified by a Modular design Superior procedure semantics support this better Implementation units should communicate through welldefined interfaces (not global variables). What is so bad about global variables? Some languages make global data easier than others Some languages make it hard to inspect internals of Modules. Others make it easier Different decisions are harder or easier to hide Algorithm Data representation Lower-level modules Policy Language Trend Towards Hiding Machine/assembly languages hide nothing C is essentially a quasi-machine-independent assembly language Early programming languages (Fortran, Cobol) had the same problem Extensive use of global data Later languages (Ada) made it possible to hide internals And made global data very awkward Modern languages (Java) make it hard to expose internals
Data Abstractions User's (client's)-eye view of the data types to be used Essentially the same as Parnas notion of a "data module" --and the notion of an "object" Cluster of "accessing primitives" / "methods" whose purpose is to provide the only mechanisms for manipulating data of a given type Problem: How to specify the semantics of these types --without specifying their implementation --internal part vs. external part Being rigorous helps separate (even slightly) different notions of an ADT from each other Languages incorporating Assertions Program code is interspersed with assert statements Assert statements define intent Assertions defined by programmer Locations identified by programmer Reactions to violations defined by programmer Assert statements may use different language semantics Usually Boolean logic Sometimes have their own private data space Assertions Specifications of intended properties of execution state e.g. relations among the values of program variables Precise specifications of Required behavior I.e. embodiments of requirements Can be used to checking relations between code and Design Requirements Acknowledge need for code to fit into overall development lifecycle But we will see other uses soon Inserted by Tester Y := current_time. <code sequence> X := current_time; run_time:= X - Y; ASSERT run_time< 2.0; <rest of code> Automatically processed into if ~(run_time > 2.0) Then Assertion_violation_handler; Assertion-Based Testing Imbed assertions with program executable code Assertions function as runtime checks Supports zooming in on internal workings of the program Examine behaviors at internal program locations while the program is executing Augments examining only final outputs Comparison: Runtime evaluation of assertions Facilities for programming reactions to violations Supports looping (rework) in the development lifecycle Tool Suites and Environments Better tools make languages more useful Can mean success or failure of a language Some major types of tools Editors Error checkers Testing aids Powerful libraries But tool integration (into environments) is key
Patterns Higher level implementation constructs Idioms (Rich and Waters, ~1985) The Gang of Four book Inspiration from real architects (C. Alexander) Idioms in common use Suggest ways that humans think/human esthetics Continuing attempt to bring languages closer to how humans think, and away from how machines work Transcend specific languages Some finding more direct support in newer languages Unit Testing Determining the characteristics and properties of a compilation unit of code In the case, the artifacts are less a focus than the activity How should unit testing be done? What process(es) should be employed? And how should the results be interpreted? Very little attention paid to the notation in which the results are expressed The Full Development Lifecycle The Code-and-Test Part Requirements Specification Architecting Implementation Designing Coding Requirements Specification Architecting Implementation Designing Coding System Test Plan Software Sys. Test Plan Integration Test Plan Unit Test Plan System Test Plan Software Sys. Test Plan Integration Test Plan Unit Test Plan System Testing Software Sys Testing Integration Testing Unit Testing System Testing Software Sys Testing Integration Testing Unit Testing Flow of control edge (the ImmFol relation) Flow of control edge (the ImmFol relation) Data Flow edge (artifacts flow from tail to head) Data Flow edge (artifacts flow from tail to head) Testing Testing is the systematic (?) search of a program's execution space The essence is comparison of behavior to intent Behavior determined by examining test execution results Intent derived (somehow) from (various) specifications Comparison typically has been done by text examination Although much more automatic testing done now Input Space Testing Program Output Space
Testing is Sampling the Input Space Key problem: What is the input space? What is the software intended to do? Subproblem: The input space is large One dimension for each program input Each dimension can have as many elements as there are legal inputs (eg. 2**32 different integers) Each input really is different How different? Which differences matter? Key Problem: How to sample from this input space? What is the input space? Specification sum_of_roots takes an arbitrarily long sequence of real numbers, and computes the sum of their square roots. The real number sequence must be ended with the number 9999.99 Implementation Program sum_of_roots; Real sum, x, r; sum := 0; Do forever input x; if x = 9999.99 then exit else r := sqrt(x); sum := sum + r; end do; print sum; end; Computing the Input Space There are ~2**32 possible different values for each input If n values are read in, then there are (2**32)**n different points in the input space The number of different input values read in is unlimited There is no limit (theoretically) to the size of the input space Some observations about the example program input space There is no real need to test every possible combination of input values Most executions behave the same But some input combinations are different Negative values will produce a failure There is a virtually limitless number of inputs that don t cause the negative square root failure A sufficiently large sequence of input values will cause an overflow failure Effective selection of test cases requires thought and care The Testcase Selection Problem Testing lets you put your program under a microscope Can examine minutiae But only for current execution To find faults you need to select test data to cause failures Testing can demonstrate the presence of faults (when suitable test cases are selected) But demonstrating the absence of faults requires knowing the behaviors of all executions But there are (virtually) infinitely many possible executions So how to sample the inputs representatively? Partitioning the Input Space Rationale: All points in the same subdomain are processed equivalently by the program But: How to determine the partition? How to know how far the equivalence holds? How to select the point(s) within each domain to use as the actual input(s)? The manual and/or requirements specification can help
Equivalence Partitioning Testing The typical approach A test of any value in a given class is equivalent to a test of any other value in that class If a test case in a class reveals a failure, then any other test case in that class should reveal the failure Some approaches limit conclusions to some chosen class of faults and/or failures Input Space Program Output Space Input Space Partitioning Black Box vs. White Box Testing SELECTED INPUTS RESULTANT OUTPUTS DESIRED OUTPUT BLACK BOX TESTING Program Input Space Divided into Domains SELECTED INPUTS RESULTANT OUTPUTS INTERNAL BEHAVIOR DESIRED OUTPUT SOFTWARE DESIGN WHITE BOX TESTING (CLEAR BOX TESTING) Structural (White Box) Testing Rigorously defined Flowgraph helps Testcase choices driven by program structure Flowgraph is most commonly used structure: Represent statements by nodes If a statement can execute immediately after another, connect the nodes representing them by an edge Every program execution sequence is a path Criteria based on coverage of program constructs All statements (node coverage) All control transitions (edge coverage) All possible paths, loop iterations (path, loop coverage) How to generate input data to do this? What exact data sets are used to force these coverages? It matters totalpay := 0.0; for i = 1 to last_employee if salary[i] < 50000. then salary[i] := salary[i] * 1.05; else salary[i] := salary[i] * 1.10; totalpay := totalpay + salary[i]; end loop; print totalpay; Different paths partition the execution space totalpay := 0.0; for i = 1 to last_employee salary[i] := salary[i] * 1.05 if salary[i] < 50000. totalpay := totalpay + salary[i] end loop print totalpay salary[i] := salary[i] * 1.10
Using Flowgraphs to Partition Input Space Requires use of static analysis (described soon) Use program branching conditions to create sets of constraints Solving constraints for each path yields the input domain that forces execution down that path Called Symbolic Execution More on this soon How to assure testing is thorough? Assure every node is covered by at least one test case (node coverage/statement coverage) Assure every edge is covered by at least one test case (edge coverage/branch coverage) Assure every loop is exercised Both by iteration and fall-through Assure all execution paths are covered Practical impossibility And there are many other criteria How to assure testing is thorough? Assure every node is covered by at least one test case (node coverage/statement coverage) Assure every edge is covered by at least one test case (edge coverage/branch coverage) Assure every loop is exercised Both by iteration and fall-through Assure all execution paths are covered Practical impossibility And there are many other criteria Best to remember that: Testing is buying knowledge Functional (Black Box) Testing Specification of Intent: Expressed explicitly Increasingly completely» Functionality, timing, accuracy, robustness,... Increasingly rigorously» Mathematics, FSA s Ideally arise directly from requirements and design specifications Specification of Behavior Tools to capture test outputs (inputs too) Comparison With automatic comparators Examples of Black Box Testing Goals Does the software do what it is supposed to do? When might it fail? How fast does it run? How accurate are the results? What are its failure modes and characteristics? What can I count on? What should I worry about? What are its strengths and weaknesses? Functional (Black Box) Testing Guidelines Result in many test cases Some test cases may satisfy many objectives Keep track of the goal(s) of each test case Changes in specification will cause changes in functional test cases Need a way to organize, use, reuse, and monitor functional testing NOTE: many of the functional testing guidelines can also be applied in structural testing (at a more detailed and formal level)
Assertions Specifications of intended properties of execution state e.g. relations among the values of program variables Precise specifications of required behavior Development of increasingly elaborate assertion languages Often used for checking relations between code and design But we will see other uses soon Inserted by Tester Y := current_time. <code sequence> X := current_time; run_time:= X - Y; ASSERT run_time< 2.0; <rest of code> Automatically processed into if ~(run_time > 2.0) Then Assertion_violation_handler; Assertion-Based Testing Imbed assertions with program executable code Assertions function as runtime checks Supports zooming in on internal workings of the program Examine behaviors at internal program locations while the program is executing Augments examining only final outputs Comparison: Runtime evaluation of assertions Facilities for programming reactions to violations Supports looping (rework) in the development lifecycle Functional vs. Structural Testing Cannot determine if software does what it is supposed to do without considering the intent a special case not handled by the implementation will not be tested unless the specification/ requirements are considered Cannot ignore what is actually done by the software program may treat one element in domain differently than stated in the specification implementation often employs use an algorithmic technique with special characteristics that are not highlighted in the specification Both functional and structural testing must be done Should use all available information in testing