Automating Requirements-Based Testing. for Hardware Design 1. Stephen T. Frezza. Dept. of Electrical Engineering. University of Pittsburgh

Transcription

1 Automating Requirements-Based Testing for Hardware Design 1 Stephen T. Frezza Dept. of Electrical Engineering University of Pittsburgh frezza@ee.pitt.edu Abstract This paper presents a methodology for automating the evaluation of partial designs using black-box testing techniques. The methodology generates black-box evaluation testsets using a novel semantic graph data model which maintains the relationships between design and requirements data. The testset is used to simulate the design module and the related requirements, thus generating a set of expected and a set of actual results suitable for analysis. I. Introduction Because requirements capture \design expectations," design quality suers when either the requirements dier from the design expectations (poor analysis) or the design does not meet its requirements (poor implementation). Design quality is dicult to assure if the relationships between the requirements and the design (traditionally referred to as traceability) are not available. To this end, we develop a unied semantic graph representation of requirements and design data, where links in the graph explicitly represent relations among the requirements and design data. Our goal is to improve design quality by utilizing these data structures to automate performance evaluation testing of individual design modules. What we mean by requirements based testing are performance evaluation tests that answer the question, \does this part of the design perform correctly?" We approximate what is meant by `correct' by evaluating (testing) the architectural design entity in question to see if its simulated performance matches that of the requirements associated with it. As our work focuses on evaluating the design with respect to an identied set of requirements and not with its internal workings, black-box testing techniques [13] are most appropriate. By automating the generation and use of these black-box tests, we expect to improve the quality of the overall design by providing feedback early in the design process on the consistency between the implemented design and the requirements model. This paper is organized as follows. Section II presents a brief review of the related research in design representation and requirements traceability, and gives an overview of our semantic graph model, which unies the requirements and design representations. Section III describes our method for automating performance evaluation using our data model. Section IV presents an example application of the method to the testing of roundo for a oating-point arithmetic unit. Section V summarizes and concludes the paper. II. Background: Representing Requirements and Design Research in the area of design representation and methodologies has supported advances in the development of Software and Requirements Engineering Environments [5, 9], Computer-Aided Design (CAD) Frameworks [3, 15] and Automated 1 This work was supported, in part, by the National Science Foundation under Grant MIP

2 Design Systems [8]. This literature identies two broad categories of design information that need to be represented in a CAD system: requirements denition (RD) data, and architectural design (AD) data. Representation work for computer hardware AD data has mostly been in the area of VLSI CAD Databases, where research has focused on the ecient representation and retrieval of design information, particularly version and equivalence maintenance [10, 16, 18]. For example, in the Version Data Model [10], the equivalence, conguration and version relationships are explicitly considered as representational dimensions of design information (see Figure 1a). Configurations Configurations Equivalences Equivalences Versions Fig. 1. (a) Version Data Model (AD) Viewpoints Versions (b) Requirements Data Model (RD) The Version Data Model is interesting in that it represents a common terminology and collection of mechanisms for representing engineering design data. Conguration relationships support the design hierarchy, equivalence relationships describe how one design description is similar to another design description for the same design object, and version relationships describe how one variant of a design entity is related to another variant of the same entity. This is a unied approach to design representation because all of the information and the relationships between dierent abstraction levels are maintained in the same database [11]. In our unied model, relationship information is explicitly maintained and available for traceability use, a feature we rely on heavily for test generation. Based on the design data model and work in developing frameworks for requirements data [5], we identify a similar set of relationships that exists for requirements data. In our requirements data model, (Figure 1b) requirements entities have four types of relationships: conguration (derived-from), equivalence (same-as), version (derived-from), and viewpoint (related-to). This model allows requirements information to be stored in dierent formats, such as Entity-Relationship or line-item, and allows representations for the connections among the RD entities. One particular equivalent form of RD data that we are interested in for test generation is simulateable requirements models. Dierent simulateable requirement modeling languages have been proposed [4, 17], and the simulateable requirements modeling language we have integrated into our system is Requirements Specication Language (RSL) [1]. A key aspect to being able to use any simulateable requirements model eectively is the ability to focus the simulation on the appropriate part of the requirements model. Here the relational links within and between the RD and AD data sets serve a key role, because they provide the means for identifying the subset of the requirements that are applicable to the design module under consideration. This is because these links provide a thread of origin from the implementation to the requirements [6], and serve as a validation that the design does indeed do what it was intended to do. In this way, these relational links are rightly termed traceability links. We identify four classes of traceability links which we support, and identify their rolls in tracing through the requirements denition and architectural design data: Rational Dependency [RD!AD]: purpose of design object tied to a particular (not-

3 null) set of requirements; normally called forward traceability links [14]. Technical Dependency [AD!AD]: dependence of one design object on another to perform/meet its requirements. This link relates to the design entities' combined ability to do the right thing - and typically encompasses the interface/connections internal to the design. These links would also include the conguration relations among the design entities as part of their traces. Contextual Dependency [RD!RD]: purpose of requirement object tied to other requirements objects, and includes the conguration relations among RD data. Can include requirements that are derived (or implicitly stated) in the environment, such as where optative descriptions imply (or rely upon) assertive descriptions within the requirements model. Sometimes classied as pre-rs traceability links [6]. Implicative Dependency [AD!RD]: addition of a particular design entity implies other requirements/constraints on the design. Typical example would be where design decisions aect/inuence the requirements denitions. Similar to contextual dependencies, these form one class of links normally called reverse traceability links [14]. Our semantic graph model is unique in that it identies the classes of relationships that need to be maintained between the requirements and design data. In each part of the model, we use links to both represent a relationship and maintain the necessary information about the relationship. The key benet of the model is highlighted in the ability to automatically generate black-box performance evaluation tests. III. Automated Performance Evaluation Black-box performance evaluation testing for a particular design module involves four steps: the identication of input classes, the selection of values for the input classes to generate a black-box testset, the generation of correct results, and the generation of the test results. There are three steps, each supported by its own sets of support code required to generate the testset need for the generation of correct results. These steps are shown as the numbered program mappings in the roundo example depicted in Figure 2. The object of these steps it to generate a Roundo Testset for the design implementation and requirements models for the roundo generation in a oatingpoint arithmetic unit (FPAU). We will pick up on this example in more detail in Section IV. SP Format SP Format Infinite Precision G & R Bits G & R Bits Rounding Modes RTNE RT+oo RT-oo RTZ RTNE RT-oo RT+oo RTZ Requirement Definitions Related RSL Statements FP ALU Roundoff Testset 1 Simulateable Roundoff Specification Correct Results FP+ Architectural Design Roundoff 2 3 Roundoff I/O Specification SRS Links to Requirements Ties to Specifications Program Mappings Fig. 2. Black-box Test Generation for FPAU Roundo

4 The rst program mapping (1) uses the trace links in the database to collect the relevant RSL requirements model related to the Roundo design module and form them into a Simulateable Requirements Specication (SRS). Similarly, a second program mapping (2) is used to generate the input/output specication for the design entity. Generating the I/O specication may not be trivial to determine, as it depends on the format of the design entity. The third program mapping (3) is the formulation of the testset for the design entity, and heuristics that implement standard black-box testing techniques [13] using information from the (design) I/O specication and the simulateable requirements specication are used to generate the input classes and the testset. The key to the testset generation procedure lies in the trace link information stored in the database. The trace links from the design inputs and the associated SRS statements serve as the basis for identifying test input classes, and for insuring that the names used in the testset can be applied to both the design and requirements simulations. Input classes consist of input ranges, determined by the data type use in the design. Simple heuristics are applied to the input ranges to determine what values to test for, e.g., for bit strings the min, min+1, mid, max-1 and max values are tried. These combinations are then checked for their control content and for redundancies to keep the generated testset from growing unnecessarily large. A mapping from the input names in the design to the input names in the SRS is needed to insure that the testset generated can be applied to both the design and the assembled SRS to generated comparable results. The next important task is that of generating the `correct' and implemented results. The correct results are generated by applying the testset to a simulation of the SRS, and the implemented results are generated by applying the testset to a simulation of the implemented design. Entity: Round Type: VHDL entity round is port( Mode: IN bit_vector( 1 downto 0); SPostNorm: IN bit; EPostNorm: IN bit_vector( 7 downto 0); MPostNorm: IN bit_vec VHDL Simulation Rules StickyPN: IN bit; Result: OUT bit_ve if (process blocks) then clk: IN bit); use end round; architecture proc of round i signal R:bit; signal L: bit; signal Carry: bit; else signal Inexact: bit; use begin Simulator Type ::= vsim Compilation ::= make_vsim Simulator ::= ${entity}_vsim ::= round_vsim Simulator Type ::= vsim Compilation ::= vcomp R <= MPostNorm(0); L <= MPostNorm(1); Simulator ::= vsim process endif; variable tempm: bit_vector(24 downto 0); variable tempe: bit_vector( 8 downto 0); variable i: bit_vector( 4 downto 0); begin wait until (CLK falling = 1 ); case Mode(1 downto 0) is when B"00" => -- Round To Nearest Even... testset.vsim_in round_vsim testset.results Fig. 3. Simulation of Implemented Roundo Design Figure 3 shows the evolution from a design entity to a generated test. Here the design entity type is matched with the rules for generating a simulation for that design entity. In the FPAU roundo example, the design entity type is VHDL [7] (VHSIC Hardware Description Language), and there are two means of generating VHDL simulators: one for process (behavioral), and one for non-process VHDL entities. Once the simulation is set up, the testset, formatted for the particular simulator (testset.vsim in) is given as the input to the simulation, and the results are collected for comparison to the 'correct' roundo SRS simulation results.

5 The presentation and comparison of the test results is important, as all discrepancies need to be highlighted, and the individual test setups made available to the designer. We do not address the interface issues, as our emphasis is on generating the information rather than presentation. IV. The Roundo Example As an example of the representation and test-generation methodology we are proposing, consider one small but important aspect of the design of a Microprocessor: the implementation of roundo in the FPAU. Our goal is to evaluate the performance of this isolated part of the design, and compare it to its requirements. Many of the requirements for roundo performance are contained in the ANSI/IEEE Standard for Binary Floating-Point Arithmetic [2]. Other requirements can be derived from standard understandings of computer arithmetic, all of which we capture in the requirements model. Figure 4 shows a fragment of the requirements database, including some line-item and RSL representations for requirements for oating-point number formats. The required number formats take on particular Sets of Values; for which Precision, Max and Min Exponent values, specic Format Parameters, etc. are dened. Requirements Definition Entities Requirement Executable (RSL) Specification Sets of Values ANSI/IEEE FP Standard Formats Basic Formats Configuration Relation Equivalence Relation Single Fmt Parameters Precision p=24 Format Parameters DATA: UNPACKED_SIGN_A. 1bit sign s LOCALITY: LOCAL. TYPE: BOOLEAN. Exponent Bias=127 exponent Format Width=32 Exponent Width=8 fraction f DATA: UNPACKED_SP_SIGNIFICAND_LENGTH. LOCALITY: GLOBAL. TYPE: INTEGER. INITIAL_VALUE: 24. DATA: SP_EXPONENT_OFFSET. LOCALITY: GLOBAL. TYPE: INTEGER. INITIAL_VALUE: 127. DATA: UNPACKED_EXPONENT_A. LOCALITY: LOCAL. TYPE: INTEGER. FILE: UNPACKED_SP_SIGNIFICAND_A. LOCALITY: LOCAL. CONTAINS: DATA: UNPACKED_BIT_VALUE DATA: UNPACKED_SP_SIGNIFICAND_BIT_POSITION. ORDERED_BY: DATA: UNPACKED_SP_SIGNIFICAND_BIT_POSITION. Fig. 4. Example of a requirements hierarchy showing conguration and equivalence relations There are four dened formats: one of which is the 32-bit Single Precision format, the parameters for which have specic values associated with particular bit elds. Also shown are the Basic Formats requirements which are comprised of eld denitions. These requirements are linked by conguration links. Figure 4 also shows a set of equivalence relations to RSL constructs, which are simulateable specication types associated with particular requirements, e.g., DATA: SP EXPONENT OFFSET, is associated with the Exponent Bias=127 requirement. In the FPAU design, these number formats are implemented in the data-types used. For example, the VHDL designer represented the exponent as an eight-bit value, and the name of the post-normalized exponent eld (one of several inputs to the roundo module) was EPostNorm. These names and their types are used to dene the ranges for the black-box input classes: mode(1 downto 0), SPostNorm, EPostNorm(7 downto 0), MPostNorm(0), MPostNorm(23 downto 1) and StickyPN

6 respectively. The names for the input classes come from the linked requirements entities, which for the roundo example are mode, sign, exponent, R, fraction, and sticky. Before testset reduction, the six roundo input classes would each have ve potential values: Min, Min + 1, Mid, Max - 1, and Max yielding 5 6 = test cases. Here mode is a control variable, and is associated with the four required FP rounding modes; sign, round and sticky are bits having exactly two values, leaving exponent and fraction to take on the ve specied values yielding a testset containing = 800 cases. These input classes and ranges are shown in Table I. Class name Values Count Notes [design name] Mode 00, 01, 10, 11 4 Enumerate [Mode] controls Sign 0, 1 2 Reduces to [SPostNorm] Min/Max Exponent , , Min, Min + 1 [EPostNorm] , 5 Mid , Max - 1, Max Round 0, 1 2 [MPostNorm(0)] Fraction , Max [MPostNorm(23..1)] , Max , 5 Mid , Min Min Sticky 0, 1 2 [StickyPN] TABLE I Reduced input test classes and values for fp roundoff This table summarizes the reduced testset for round. Using these values to generate the simulation input, we calculated the correct results by hand. Once completed, we expect our system to generate the correct results by simulating the generated SRS with the testset as mentioned. The following presents parts of the test results for the VHDL design entity round: The design simulation output variable is result, and the correct results are shown as correct result =... output line. The simulator used was vsim [12], which is an event-level simulator. Number Mode SPostNorm EPostNorm MPostNorm(0) MPostNorm(msb) StickyPN ======================================================================== Case 18 RTNE min min max min max correct result = result = time = = ns Case 19 RTNE min min max max min correct result = result = time = = ns Case 20 RTNE min min max max max correct result = result = time = = ns Case 21 RTNE min min+1 min min min correct result = result = time = = ns The summary results for cases 19 and 20 show discrepancies between the correct

7 results (correct result =...) and the implemented results (rresult =...). As it turns out, this error was caused by some extraneous code in the behavioral design for roundo, which was determined to be the cause of four other errors detected in 800 tests (test cases 59, 60, 159, and 160). Once the extraneous code was removed, subsequent use of the testset discovered no more errors. V. Conclusion In this paper, we have presented an outline for a unied data model for representing linked requirements and design data. Based on this data model, we presented a methodology for automating performance evaluation testing using black-box testing techniques. We also presented the details of an example showing the generation of black-box performance tests for the roundo of a oating-point adder. References [1] Alford, M. Software Requirements Engineering Methodology (SREM) at the Age of Eleven: Requirements Driven Design. In Modern Software Engineering: Foundations and Current Perspectives, ch. 11, pp. 351{377. Van Nostrand Reinhold, [2] American National Standards Institute and the IEEE Standards Board, New York, NY. IEEE Standard for Binary Floating-Point Arithmetic, ANSI/IEEE Std [3] Armstrong, R. and Allen, J. FICOM: A framework for Incremental Consistency Maintenance in Multi-Representation, Structural VLSI Databases. In IEEE/ACM Int. Conf. on Comp. Aided Design (ICCAD `92), pp. 336{343, November [4] Davis, A. A Comparison of Techniques for the Specication of External System Behavior. Communications of the ACM, 31(9), September [5] Finkelstein, A., Kramer, J., Nuseibeh, B., Finkelstein, L., and Goedicke, M. Viewpoints: A Framework for Integrating Multiple Perspectives in System Development. Int. J. of Software Engineering and Knowledge Engineering, 2(1):31{58, March [6] Gotel, O. and Finkelstein, A. Modeling the Contribution Structure Underlying Requirements. Proc. of the 1st Int. Conf. on Requirements Engineering (ICRE), pp. 94{101, April [7] IEEE Press, New York, NY. IEEE Standard VHDL Language Reference Manual, IEEE Std [8] Jacome, M. and Director, S. Design Process Management for CAD Frameworks. Technical Report EDRC , Engineering Design Research Center (EDRC), [9] Johnson, W. L., Feather, M., and Harris, D. Representation and Presentation of Requirements Knowledge. IEEE Trans. on Software Engineering, 18(10):853{869, October [10] Katz, R. H. Toward a Unied Framework for Version Modeling in Engineering Databases. ACM Computing Surveys, 22(4):375{408, December [11] Kollaritsch, P., Lusky, S., Matzke, D., Smith, D., and Stanford, P. A unied Design Representation Can Work. In 26th ACM/IEEE Design Automation Conf., pp. 811{813, June [12] Martello, A. and Levitan, S. A VHDL Design Environment. SIGDA Newsletter, 20(3):52{67, December [13] Myers, G. J. The Art of Software Testing. John Wiley & Sons, [14] Ramesh, B. and Edwards, M. Issues in the Development of a Requirements Traceability Model. In Proc. of the IEEE Int. Symp. on Requirements Engineering, pp. 256{259, January [15] Siepmann, E. A Data Management Interface as part of the Framework of an Integrated VLSI Design System. In IEEE/ACM Int. Conf. on Comp. Aided Design (ICCAD `89), pp. 284{287, November [16] Singhal, A., Parikh, N., Dutt, D., and Lo, C.-Y. A Data Model and Architecture for VLSI/CAD Databases. In IEEE/ACM Int. Conf. on Comp. Aided Design (ICCAD `89), pp. 276{279, November [17] Webster, D. E. Mapping the Design Information Representation Terrain. IEEE Computer, 21(12):8{23, December [18] Wu, A., Hadley, T., and Gajski, D. An Ecient Multi-View Design Model for Real-Time Interactive Synthesis. In IEEE/ACM Int. Conf. on Comp. Aided Design (ICCAD `92), pp. 328{331, November 1992.