HIGH-LEVEL TEST GENERATION FOR HARDWARE TESTING AND SOFTWARE VALIDATION

Transcription

1 HIGH-LEVEL TEST GENERATION FOR HARDWARE TESTING AND SOFTWARE VALIDATION 0. Goloubeva, M. Sonza Reorda, M. Violante Politecnico di Torino, Torino, Italy, it Abstract It is iioiv coninion for design teams to develop systems where hardware and software components cooperate; they are thus facing the challenging task of validating and testing systems where hardware and software parts exist. In this paper a high-level test generation approach is presented, u+,hich is able to produce input stimuli that can be fruitfully exploited for test and validation purposes of both hardware and sofh$3are coniponents. Experinteiital results are reported showing that the proposed approach produces high quality vectors in temis of the adopted metrics for hardware and sofh%,are faults. 1. Introduction In the last years, new techniques have been developed to integrate an entire system on a single chip, resulting in the new design paradigm hown as System-on-Chip (SOC). SOC products represent a real challenge not only from the manufacturing point of view, but also when design issues are concerned. Designers have thus to face the challenging task of validating and then testing systems where hardware and software components cooperate. While the design practice is quickly moving toward higher levels of abstraction thanks to the adoption of system-level design tools, test issues are still mainly considered only at the lowest levels of abstractions, when a detailed description of the design is available. For the hardware components, test is typically addressed at the gate level for test sequence generation and at the register transfer (RT) level for design for testability structure insertion. As far as software components are concerned, the exploitation of formal verification techniques has been advocated by some authors [l] as a viable solution for assessing their correctness, hut its widespread adoption is limited by the overhead, both in terms of expertise and resource, it implies. Several approaches have been proposed to overcome these limitations. Symbolic evaluations 121 [3] consists in first assigning symbolic values to variables; the paths composing the program control flow graph are then traversed, and a list of symbolic representations of each condition predicate along the paths is recorded. Through the analysis of the obtained constraints it is then possible to determine if paths are executable as well as the conditions for their activation. Simpler approaches are those based on path testing [4] and test coverage metrics [5], where the number of executed paths, branches and statements are counted, and those based onfunctional testing [5][6], where software components are considered as black boxes and the outputs they produce in response to a set of input stimuli are checked for conformity with the expected behavior. Finally, an approach inspired to those adopted in the hardware test community is mutation testing [7][8]. When adopting mutation testing, faults are introduced into a software component by creating many versions of the component, each of which contains one fault. A faulty component is thus a mutanf of the original one. The obtained components are then executed under a given set of input stimuli to observe which faults produce wrong results. Usually, the input stimuli used during the mutation testing process are either provided by designers or they are randomly generated within a Monte Carlo simulation procedure. As a result, the input stimuli do not guarantee complete coverage of all the mutants possibly affecting the tested software component. Today, the introduction of system-level design tools has unified the process of specification of software and hardware components of Systems-on-Chip. As an example, the users of co-design tools start the design process by just describing the system behavior, exploiting a system-level specification language such as SystemC [9], while neglecting which part of the specification will be implemented in hardware and which one in software. In this scenario, provided that a test generation process is available to deal with behavioral specifications, the automatic generation of a single set of input stimuli suitable for testing both hardware and software components becomes possible. Starting from the preliminary work we presented in [lo], in this paper we explore the feasibility of a validatiodtest approach based on a test generation algorithm that works on high-level descriptions and that is able to provide input stimuli (in the following called test vectors) useful for testing both hardware and software components. For the purpose of this paper, we target purely behavioral system descriptions, where the behavior of each SOC component is coded resorting to the SystemC /03/$ IEEE 143

2 specification language. Complex systems embedding multiple components are also considered. The approach we propose exploits a high-level fault model for driving the test generation procedure, which is performed by an algorithm based on a heuristic search. Test vectors can be generated without any knowledge about the analyzed system except its behavior. The results gathered on several benchmark systems show that the generated vectors can be fruitfully used for testing the hardware implementation of the given behavior, as well as for validating the software one. This result allows to look at the area of high-level test stimuli generation in a new way, since it shows that generation can be performed not only without the knowledge of any implementation detail, but even irrespectively of what is implemented in hardware and what in software. The remainder of the paper is organized as follows. In -section 2 we review the state-of-the-art in high-level test vector generation and fault models. In section 3 the proposed method of the test vectors generation is presented. Section 4 reports the results we gathered on some benchmarks and finally section 5 draws conclusions. 2. Previous works The availability of an effective behavioral-level test generation process mandates the definition of suitable fault models and test generation algorithms supporting them. As far as the fault model is considered, the following characteristics should be met. On the one hand, the fault model must he applicable to behavioral specifications for example composed of variable assignments, arithmeticilogical operations among variables and control flow instructions. On the other hand, the fault model should be representative of faults that can affect either the hardware or the software components of SOCs. Several high-level fault models can be found in literature, which are inspired by those known in the software-testing [5] domain, and that extend them to cope with hardware descriptions. The state-of-the-art of high-level fault models is described in [ 111 where bitfailures and conditionsfoilures are used to model faults affecting the memory elements and the control logic of hardware components, while only their behavioral specifications are known. As far as the test generation algorithm is considered, several approaches have been proposed in the past that are able to maximize the coverage figure of the adopted highlevel fault model. Most of them are able to generate test patterns of good quality, sometimes comparable or even better than those of gate-level ATPG tools. However, lacking general applicability, these approaches are still not accepted by the industry. The different approaches are based on different assumptions and on a wide specuum of distinct algorithmic techniques. Some are based on extracting from a behavioral description the corresponding control machine [I21 or the symbolic representation based on binary decision diagrams [13], while others also synthesize a structural description of the data path [14]. Moreover, other approaches try to combine high-level information about the control machine and the data-path [15]. Some approaches rely on a direct examination of the HDL description [ 161, or exploit the knowledge of the gatelevel implementation [17]. Some others combine static analysis with simulation [ 18][ 191. The common denominator of the alrkady proposed approaches is that they aim at generating input stimuli targeting faults affecting hardware components, only. They are indeed intended to be exploited in design flows based on behavioral synthesis, where the system is a purely hardware one. Furthermore, the goodness of the produced input stimuli is evaluated as the gate-level stuck-at fault coverage they attain. 3. High-level test generation environment In this section we describe the high-level test generation environment we developed. In panicular, section 3.1 describes our assumptions on the considered systems, section 3.2 discusses the adopted, fault models, while section 3.3 presents the test generation algorithm we implemented Assumptions For the purpose of this paper we assume that a system is a network of components described in a purely behavioral fashion. Variable assignments, arithmetic and logical operations among variables and control flow instructions are used to describe how the components react to a set of input stimuli to produce the corresponding output values. Communication among components is assumed to be asynchronous, based on events. Given a set of input stimuli arranged as a sequence of input vectors, we assume that a new vector is applied to the system inputs only when the system is in a steady state, i.e., when the previous vector has been evaluated and the corresponding output values have been produced High-level fault models Several high-level fault models are available in the literature that can be used for assessing the goodness of test vectors while working at abstraction levels higher than the gate-level one. For the sake of this paper, we considered the high-level fault models described in [ 111 which provide an accurate estimation of the test capabilities of input vectors while working on behavioral descriptions. The considered fault models are: I Bit coverage: each bit in every variable, signal or port in the system can be stuck to zero or one. The bit coverage measures the percentage of stuck-at bits that are propagated on the system outputs by a given test sequence. 144

3 Condition coverage: each condition can be stuck-at true or stuck-at false. Then, the condition coverage is defined as the percentage of stuck-at condition that is propagated on the system outputs by a given test sequence. In order to fruitfully exploit the aforementioned highlevel fault models within a test generation tool, we developed a high-level fault simulation environment, which implements the Saboteur [20] approach through a two-step process: 1. The behavioral system description under analysis is first instrumented by adding suitable statements that fulfill two purposes: 2. a. They alter the behavior of the system according to the supported fault models. b. They allow observing the behavior of the system to gather meaningful statistics (in particular, they provide access to the contents of all the variables in the system description). During this phase, the list of faults to he considered during fault simulation is also computed and stored. A given set of input vectors is applied to the inputs of the system description resorting to the adopted simulation environment. During the execution of the system description, a preliminary simulation is performed without injecting faults, and the output trace of the system is recorded. Then, each fault in the previously computed fault list is injected and faulty output trace is recorded. By comparing the faulty trace with the fault-free one, we then compute the high-level coverage figure the vectors attain High-level lest generation algorithm The test generation algorithm we developed implements a Random Mutation Hill Climber (RMHC) algorithm, whose pseudo-code is reported in the figure I. A RMHC is a Hill Climber that, given a current solution, evaluates neighhor solutions in a complete random order until an improvement is found. When an improvement is found, the process is iterated over the new solution. The process is repeated until a given stopping condition is met. The adopted stopping condition corresponds to performing a given number of iterations without improvements. In our algorithm a solution is a sequence S of test vectors; each test vector is applied to the system inputs according to the assumptions stated in section 3.1. Starting from an initial randomly generated solution S, a ned solution S is computed by applying a random mutation operator. This operator supports three types of mutations: it complements one randomly selected bit within a randomly selected vector of S, it increases the number of vectors in S by adding a randomly generated vector in a randomly selected position in the test sequence, or it decreases the number of vectors in S by removing a randomly selected vector in the sequence. The new solution S is accepted if and only if it increases the goodness of the previous solution S, HLTGISI while1 stopping condicion not met I t S = apply_random_mutationlsl if (Fitness1 IS ) > Fitness1 IS1 I I I 1 I Fitnessl (S l == Fitness1 (SI I && I Fitness2 IS ) > Fitness2 IS1 I I s = S if( new faults are detected I save-solution (SI Figure 1. HLTG algorithm In the HLTG algorithm, the goodness of a solution is defined by two functions: 0 Fitnessl (S) = Coverage(S) FitnessZ(S) = NS(s) where: Coverage(S) is the sum of the bit coverage and condition coverage as measured by the high-level fault simulator described in section 3.2. NS (S) is the number of different states the system traverses during the evaluation of the sets of vectors. The state of the system is defined as the content of every variable in the system at the end of the evaluation of one input vector. This figure is computed by exploiting the information provided by the highlevel simulator. The new solution S is accepted if Fitnessl (S ) > Fi tnessl (9, where S is the current solution. In the case Fitnessl(S 1 =Fitness1 (9, S is accepted if Fitness2 (S ) >Fitness2 (S). This assumption guarantees that a new solution is accepted only if it does not reduce the number of faults the previous solution detects. 4. Experimental results The main purpose of the performed experiments was to assess the goodness of the vectors our high-level test generation process is able to provide. For this purpose, we considered six benchmarks coded in SystemC, whose characteristics are described in table 1, where the number of SystemC lines, number of variables and operations used in the system description, number of primary inputs (PIS) and primary outputs (POs) are reported. 145

4 The fnst five benchmarks are composed of one component only, while the last one is composed of two components. Each component in the system is an instance of the SC-MODULE class whose behavior is described through the SC-METHOD primitive. The fnst f0w benchmarks inspired by those in the HLSynth 92 benchmarks suite and implement data-processing algorithms, while LRU implements a control-intensive algorithm with few data-processing operations. Finally, the last benchmark implements both data-processing and control-intensive algorithms. we run HLTG on the described benchmark, and we obtained a set of test vectors to be used for both hardware and software testing. Table 1. Benchmark characteristics I I Lines I Variables I Operations I PI I PO I Starting from the SystemC description of each benchmark we derived two implementations: A hardwure one, obtained by running the SystemC compiler. A sofhvure one, obtained by hand-coding the benchmarks in standard C language. Each component of the high-level model is translated to a procedure that is called by the main program, which operates as a very simple task scheduler. faultsim Behavioral coverage Figure 2. Simulation of HLTG vectors We implemented a prototypical version of the environment (named HLTG) that amounts to about 2,000 lines of C code. Following the flow described in figure 2, coverage & coverage Figure 3. Simulation of testgen vectors To measure the goodness of the obtained vectors we adopted two metrics: for the hurdwure implementation we adopted the single stuck-at fault coverage, and we resorted to the Synopsys faults im tool for gate-level fault simulation. for the sofhvure implementation we adopted software mutant coverage, and we resorted to an in-house developed tool (called msim) to measure it. The following mutants, taken from the current practice in software testing [7],[8] were considered a. Replacement of arithmetic operators h. Changing of the value of constants C. Replacement of relational operators d. Replacement of variables in operations and assignments e. Replacement of logical operators f. Deletion of operands from arithmetical operations. According to the authors of [7] and [PI, these mutants are representative of typical errors found in software; therefore, we assumed that any input sequence able to maximize the mutant coverage is thus likely to highlight typical errors affecting software components. The obtained results are reported in table 2, where the stuck-at fault coverage (FC) and the mutant coverage (MC) are shown, as well as the test length (Len) of the generated stimuli and the time for running the HLTG tool (CPU). 146

5 Table 2. HLTG results I I Len I CPU I MC t FC I The number of produced HLTG vectors is usually higher than that coming from the gate-level ATPG. This result is not surprising, due to the limited complexity of the high-level algorithm, where no efforts have been yet devoted to reducing the test length. 5. Conclusions All the experiments have been performed on a Sun Enterprise 250 machine equipped with two processors running at 400 MHz and with 2 Ghytes of RAM. For comparison sake, we exploited the flow shown in figure 3 to compute a set of test vectors at the gate-level generated by the Synopsys testgen tool. Being interested in evaluating the effectiveness of the HLTG vectors in detecting all the possible and subtler faults (like the ones affecting circuit gates of hardware components that can be only partially modeled at the high level) we let testgen to execute the test generation algorithm on the whole fault list, considering faults on gates and flip-flops as well. We then fault simulated the vectors with msim to measure the achieved software mutant coverage. The attained results are shown in table 3. Table 3. Tes tgen results PIease note that we simulated the vectors computed by testgen and HLTG resorting to the same tools and exploiting the same hardware and software implementations: the attained coverage figures are thus comparable. By comparing the figures in tables 2 and 3 we can observe that: Although very simple, HLTG is able to produce high quality test vectors. For 4 out of 6 benchmarks the coverage results the HLTG vectors provide are equal or better than that of vectors generated by a commercial gate-level ATPG, as far as both mutant and stuck-at fault coverage are considered. For all the considered benchmarks, HLTG is able to finish the test generation procedure in an amount of time 3 to 60 times lower than that spent by testgen. By addressing test on behavioral descriptions, and thus neglecting all the details that gate-level descriptions imply, we can effectively reduce the test generation time. This paper describes a preliminary analysis devoted to assess what can reasonably be done in terms of test stimuli generation starting from a high-level behavioral description of a system, only. We proposed a fault model suitable to be adopted on such a kind of descriptions, and described a simple algorithm for generating test stimuli from SystemC descriptions. We then performed a set of experiments, which show that the generated stimuli own surprisingly good properties in terms of both validation and test purposes, when compared to stimuli generated at the gatelevel with traditional test tools. Obviously, we do not believe that high-level generated vectors are the ultimate solution for the test and validation problem; indeed, they are often not likely to be able to cover all the possible faults in a system, mostly due to the existence of faults that can hardly be modeled at abstraction levels higher than the gate-level one. Moreover, despite the availability of valuabie vectors starting from the early design phases, the current test approach, i.e., full scan, is not likely to change in the near time. Nevertheless, this paper shows that vectors generated from high-level behavioral descriptions can be fruitfully exploited in the test and validation process: high-level generated vectors can be of great help in reducing test generation costs thanks to the exploitation of abstract descriptions that require less CPU resources for simulation and test generation. The authors are currently working to better understand the possibilities which this preliminary work opens. References 1. S. Edwards, L. Lavagno, EA. Lee, A. Sangiovannj Vincentelli, Design of embedded systems: formal models, validation, and synthesis, Proceedings of IEEE, Vol. 84, No. 3, March 1997, pp J.S. King, Symbolic execution and program testing, Communications of the ACM, No. 19, 1916, pp P.D. Coward, Symbolic execution and testing, IEE Colloquium on Software Testing for Critical Systems, 1990, pp. 2/1-2/3. 4. W.E. Howden, Reliability of the path analysis testing strategy, IEEE Trans. on Software Engineering, Vol. 2, No. 3, 1976,pp B. Beizer, Software Testing Techniques, (2nd ed.). Van Nostrand Rheinold, New York,

6 W.E. Howden, Functional program testing, IEEE Int l Conference on Software and Applications, COMPSAC 78,1978, pp R.A. DeMillo, D.S. Guindi, W.M. McCracken, A.J. Offutt, K.N. King, An extended overview of the Mothra software testing environment, IEEE Workshop on Software Testing, Verification, and Analysis, 1988, pp J. Offutt, A practical system for mutation testing: help for the common programmer, IEEE Int l Test Conference, 1994, pp SystemC User s Guide, Synopsys, CoWare, Frontier Design. G. Jervan, Z. Peng, 0. Goloubeva, M. Sonza Reorda, M. Violante, High-level and hierarchical test sequence generation, IEEE Int. Workshop on High Level Design Validation and Test, October 2002, pp F. Ferrandi, F. Fummi, D. Sciuto, Test Generation and Testability Alternatives Exploration of Critical Algorithms for Embedded Applications, IEEE Trans. on Computers, Vol. 51, No. 2, February 2002, pp D. Monndanos, J.A. Abraham, Y. Hoskote, A Unified Framework for Design Validation and Manifacturing Test, Proc. IEEE International Test Conference, 1996, pp F. Ferrandi, F. Fummi, D. Sciuto, Implicit Test Generation for Behavioral VHDL Models, Proc. IEEE International Test Conference, 1998, pp F. Fallah, P. Ashar, S. Devadas, Simulation Vector Generation from HDL Descriptions for Observability- Enhanced Statement Coverage, Proc. Design Automation Conference, 1999, pp M. Boschini, X. Yu, F. Fummi, E. M. Rudnick, Combining symbolic and genetic techniques for efficient sequential circuit test generation, IEEE European Test Workshop, 2000, pp S. Chiusano, F. Como, P. Prinetto, Exploiting Behavioral Information in Gate level ATPG, JETTA The Journal of Electronic Testing: Theory and Applications, Kluwer Academic Publishers, No. 14, 1999, pp E.M. Rudnick, R. Vietti, A. Ellis, F. Como, P. Prinetto, M. Sonza Reorda, Fast Sequential Circuit Test Generation Using High level and Gate level Techniques, Proceedings IEEE European Design Automation and Test Conference, 1998, pp M.B. Santos, F.M. Goncalves, I.C. Teixeira, J.P. Teixeira, RTL-based functional test generation for high defects coverage in digital SOCs, Proc. IEEE European Test Workshop, 2000, pp F. Como, M. Sonza Reorda, G. Squillero, G., High level Observability for Effective High level ATPG, Proc. 18th IEEE VLSI Test Symposium, 2000, pp J. BouC, P. PCtillon, Y. Crouzer, MERSTO-L A VHDL-Based Fault Injection Tool for the Experimental Assessment of Fault Tolerance, Proc. Int. Symp. on Fault Tolerant Computing, FTCS-28, 1998, pp