EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores

Transcription

1 EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores Haihua Shen, Yunji Chen, and Jing Huang Institute of Computing Technology Chinese Academy of Sciences Beijing, China {shenhh, cyj, Abstract: Core-based system-on-chip (SoC) design is quickly becoming a new paradigm in electronic system design due to the reusability of IP cores. However, the validation of IP cores is the most time consuming task in the design flow. This paper presents EmGen, an automatic test-program generation tool designed for embedded microprocessor cores. EmGen provides an configurable formal specification model with heuristic knowledge, which can generate test programs according to different configuration of microprocessors architecture, a test generation scheme based on heuristic algorithms, which can efficiently provide instructions in test programs, and validation testbenches, which support simulation with generated test programs automatically and check the equivalence of microprocessors and the specified instruction reference model. EmGen is currently in use at ICT for the verification of embedded microprocessor cores. Experiments results show that EmGen can improve verification process and cut down skilled manpower obviously. 1 Introduction Today, microprocessor vendors are suddenly clamoring to take the lead in offering reusable IP cores ranged from North Bridge to 32-bit microprocessor cores a trend that promises to shorten the time to market and fill the design productivity gap due to development of deep sub-micron technologies. Compared with general multimilliongate processors, embedded microprocessor cores pay more attention to low-cost and other attributes such as flexibility and reconfigurability. Traditional design verification comprises a large portion of the effort in designing a processor [1]. Simulationbased verification tries to uncover errors of design by detecting circuits faulty behavior when deterministic or pseudo-random simulation vectors are applied. Many practices involve in technologies to improve test generation and coverage analysis [2][3][4]. Some papers are more concerned with comparison among those advanced techniques within existing strategies of test generation. It indicates that using random test bench to perform the directed testing is strongly recommended [5]. The primary motivation of our work is to take advanced test generation technologies into practice to adapt to the feature of embedded microprocessor cores verification. Although it presents a set of new techniques to realize recommendation trends, the main contribu- Z. Wu et al. (Eds.): ICESS 2004, LNCS 3605, pp , Springer-Verlag Berlin Heidelberg 2005

2 EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores 529 tion of our work is not in theory, but in describing how to translate this theory into practice in a practical way, a task that is far from trivial. This paper presents an automatic test generation tool named EmGen, which is able to induce test programs according to design configuration for simplifying the process of verification. EmGen is based on a previously developed configurable formal specification model with heuristic knowledge that can specify request from highly directed tests to completely random ones according to the requirement of different microprocessor cores configuration and coverage metrics, a reference instruction set simulator, which can provide accurate reference results for DUT verification in executable framework, a test generation scheme with heuristic algorithms, which can efficiently provide appropriate instructions based on the configurable formal specification model, and validation testbenches, which support simulation with generated test programs automatically and check the equivalence of microprocessor cores and their reference model [6]. EmGen s uniqueness, when compared to general-purpose verification tools in stimuli generation [7] [8], lies in the fact that both the generation schemes and the validation testbenches it provides are oriented at the characteristics of flexible and reconfigurable embedded microprocessors by high-level test program generation. It has been taken into practice in the Institute of Computing Technology of the Chinese Academy of Sciences for the verification of a 32-bit embedded microprocessor core. The remainder of this paper is as follows. In Section 2 & 3, we briefly describe the configurable formal specification model and the heuristic test generation algorithms used in EmGen. Then we present the validation environments and the framework of EmGen in Section 4. Finally we present experimental results in Section 5 and give some conclusions in Sections 6. 2 Configurable Formal Specification Model Given an RT-level description of a microprocessor core, simulation-based verification requires test programs and a simulator to simulate its execution. The goal we design EmGen is to generate test programs automatically and effectively according to the specified generation rules that describe the requirement of the configurable microarchitecture of embedded microprocessors and the verification plan precisely. Test generation rules are hard to specify correctly, and yet are often critical and beneficial that their specifications are correct, complete and unambiguous. Formal specifications, based on using multiple constraints to collectively define the behavior of test generation, promise to meet all requirements. Our formal specification is summarized to four style rules. The adjustability rule requires the instructions to be generated fully adapting to different microarchitecture configuration of embedded microprocessor core. Configurable microprocessor core is an emerging technology that takes the high performance of many different asics or application specific standard products (assp) into an application tailored embedded microprocessor core [9]. General options of embedded microprocessor core configuration can be the size and organization of cache & TLB, the usage of floating point units and multimedia units, the optional bus

3 530 Haihua Shen, Yunji Chen, and Jing Huang interfaces of microprocessor core, the pipeline partition according to the requirement of frequency and so on. According to the adjustability rule, the specification should initialize all parameters in relation to the configuration of microprocessor, as well as parameters of test program length and random seeds. Constraints on microarchitecture configuration of embedded core are written in the following forms:. Config parameters TLB 1 FPU 0 MMX 1 ICACHE 2 DCACHE 2. End of config.pipeline stages ALU_WB_STAGE 1 TLB_STAGE 1 MUL_WB_STAGE 1 RENAME_STAGE 0 FMUL_STAGE 3. End of pipeline stages The random rule requires the instructions with or without arrangements to be generated randomly with configurable ratio. Based on the random rule, instruction specifications should be written with appropriate ratio. The flexibility rule requires the instructions to be generated in configurable styles from pseudorandom groups with different seeds to totally directed sequence with strict constraints. Separated by operands, allowed instructions are divided into four groups. I type includes instructions with only two src operands and dst operand in registers. M type includes instructions that access memory resources. J type & B type are all instructions whose target is to change program counters. The difference between them is that instructions in B type have branch conditions and those in J type have not. Function I/M/J/B (instruction var) is defined to find if the instruction var is in I/M/J/B type. The constraints should be written according to the following reasoning: prec(i M J B) R src1 R src2 R dst prec( I M J B) (R base ((R src2 R dst ) (R src2 R dst ))) (M str M end ) prec( I M J B) T target prec( I M J B) R src1 R src2 T offset The prec construct allows the operands of an instruction with expressed precondition to be limited by the expression of constraining logic. The divisibility rule requires each constraint to constrain only the behavior of one instruction component. Equivalently, because the constraining part is isolated from each other, the rule requires the consequent to contain only one or a scope of values for one field. The four styles above are powerful enough to specify the requirement of generation from random to totally direction, see [6] for more detailed information on the model used here. Although abiding by the style rules may seem restrictive, it promises many benefits. For example, the specification is easier to maintain. Constraints can be added or removed and independently modified. It is also believed that it is easier to write and debug. Since most existing languages are already written as a list of rules, the translation to this type of specification requires less effort and results in fewer opportunities for errors.

4 EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores Test Generation Process For a given specification model, EmGen s generation process can be divided into two layers: traversing the internal specification statement tree and generating the instructions on the basis of specification. It is easy to do the statement tree travel in a hierarchical manner. But generating the instructions is more difficult because the generation process interleaved with the execution of Reference Instruction Set Simulator must maintain the intermediate processor states dynamically. The intermediate processor states embedded in all kinds of general registers, control registers, statement registers and memories are always limited by realistic resources in practice. There are several factors in contradiction with each other that may bring on difficulties in generation process, such as the limitation of realistic memory addresses with the supposed large size of idealistic virtual memory addresses. To solve these problems, we present two algorithms named the dynamic boundary check (DBC) algorithm and the scan TLB (ST) algorithm. The DBC algorithm handles the random branches targets to achieve expected quantities and quality of generation according to verification plan. Then the ST algorithm is used to generate enough efficient instructions under the limitation of memory address space in validation environments. 3.1 The Dynamic Boundary Check (DBC) Algorithm The main idea of DBC algorithm is to change the direction of instruction flows whenever they meet a boundary, carefully select the branch targets to avoid the boundary and delete the dead branch whose target is breaking the rules or terminate the generation process if it does not belong to the random freely group. The DBC algorithm is implemented by following steps. Step 1: Generate a new instruction according to the constraints of configurable specification. Judge if the instruction is a branch or jump. If the instruction is a branch, then go to Step2. Else go to Step3. Step2: Search if there is an appropriate target pc not in pc stack following all the constraints in specification and not breaking the boundary in pc stack. If the correct target pc cannot be found, then turn to Step4, otherwise, Step5. Step3: Check the boundary in pc stack. If the instruction meets the boundary in pc stack, go to Step6; else transfer the instruction to RISS and go back to Step1. Step4: Check if the instructions belongs to random freely group. If it is random freely, then cancel this instruction and turn back to Step1 to generate a new instruction instead; else print alert and terminate the generation process. Step5: Select an appropriate target pc for the instruction and transfer the instruction to RISS. Then go on the generation process in Step1. Step6: If the instruction is in a sequence, terminate the generation process and pop up an alert. Else insert a directly jump instruction behind the instruction to avoid the boundary and go back to Step1. A case of instruction flows changed by the DBC algorithm is shown in Fig. 1.

5 532 Haihua Shen, Yunji Chen, and Jing Huang Fig. 1 A case of instruction flows changed by DBC algorithm The results of experiments show that the DBC algorithm can improve the generation process obviously (see Fig. 3 in section 5). 3.2 The Scan TLB (ST) Algorithm It is important for EmGen to generate and control all the exceptions and interrupts tightly according to verification plans. In all kind of exceptions, TLB miss happens frequently because of the limitation of TLB pages. To reduce the complexity of testbench, the maxim number and size of TLB pages are always predefined. So it is very important to generate instructions in the limitation of TLB to avoid unexpected TLB miss. The ST algorithm is presented to solve this problem. Main idea of ST is dynamically initializing the TLB whenever an instruction with a pc in new page or a memory access with a new page address happened. It is necessary to keep a TLB stack and push the TLB page to TLB stack whenever a new TLB page is distributed. When all the TLB entries are distributed, new instructions program counter and memory access is limited to the old blank TLB. The ST algorithm is described as following: Step1: Initialize a TLB page according to the predefined first pc in the configure file. Step2: When generating a new instruction, search the pc in TLB stack. If the instruction is on a page distributed, do nothing to TLB stack. Otherwise insert the TLB entry to the stack. When TLB stack filled, go to Step 5. Step3: Check if the new instruction is a branch or jump. If the instruction generated in Step2 is a branch or jump, then check if the page that contains the branch target is in TLB stack. If in the stack, do nothing. Otherwise check if the TLB stack filled. If filled, regenerate the same instruction with different target by several times (Retry times can be predefined or changed dynamically in progress according to generation plan). If there is an appropriate target whose page entry is already in TLB stack, then do nothing. Else go to step 5. If the TLB stack is not filled, insert the page that contains the instruction target into TLB stack. Step4: Go back to Step1. Step5: Terminate the generation process because of the TLB miss.

6 EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores 533 The ST algorithm acts as supplementary for the DBC algorithm to improve generation process in EmGen. Experiment results show that the required number of instructions can be generated easily according to the configurable specification by means of DBC and ST algorithms (see Figure 3 in Section 5). 4 The Sketch of Validation EmGen s modeling framework contains several components: an instruction library, a configurable formal specification model with a parser, a test generator, a reference instruction set simulator (RISS), a simulator with the validation environment, an RTlevel design under test (DUT), and a result compare logic. The overall architecture is shown in Fig.2. All microprocessor instructions are described in an instruction library. The configurable formal specification model includes the configurable formal specification described above and a parser, which can parse the specification to internal data structures. Test generator selects instructions from library according to the configurable formal specification, which can be optimized by the corresponding coverage metrics. Besides, a validation environment, which supports simulation with generated test programs and checks the equivalence of microprocessor cores and their reference model automatically, is also provided by EmGen. By means of the validation environment, EmGen can judge whether the results of the test runs are correct or not. Configurable Specification Model Validation Environment Instruction Library Parser DUT RISS Test Generator Real Memory & Reg Virtual Memory & Reg Results Compare Logic Bugs?! Fig. 2. The sketch of EmGen 5 Experimental Results To demonstrate EmGen on a meaningful design, we chose a 32-bit embedded RISC microprocessor core based on MIPS instruction set with about 0.48 million gates of logic as DUT for verification. The optional configurations of the microprocessor core are 2-Way Set Associative I-cache & D-cache, 4-Way Set Associative I-cache & D- cache, no Cache & TLB, the floating point units and multimedia units. Simulations

7 534 Haihua Shen, Yunji Chen, and Jing Huang were run on Intel Pentium 4 2.8GHz HyperThreading system with 2G of main memory. Verification results: Before presented approach is taken into practice, the DUT has been verified for more than two months. Stimuli include popular benchmarks (Whetd, Dhrystone, Paranoia), real operation systems (Linux) and many manual test programs. Using the random test programs provided by the methodology, two previously unreported bugs have been found in the DUT. The process of finding bugs is now nearly automated and much easier since result compare logic can give direct location of errors. Performance results: Performance issues, such as speed and memory usage, do not pose to be problems because of the hardware improvement. So we are free to focus on generating interesting simulation inputs. It has been mentioned above that effects of generation process is influenced by resources limitation. Predefine an address space limitation available in test program generation to be 64x8Kbytes. Our experiments show that the maximum number of instructions generated by DBC&ST algorithms is an order of magnitude larger than it was without DBC&ST algorithms (see Fig. 3). Fig. 3 a) Ability of random test generator without DBC&ST Algorithm b) Ability of random test generator with DBC&ST Algorithm Coverage results After running more than 55,000,000 instructions generated, the average code coverage of the design reaches 73.8% of line coverage, 68.9% of condition coverage, 59.78% toggle coverage, which is hard to be improved by increasing more random instructions. It is obviously that the likelihood of generating more effective events so unintentionally is low, though the EmGen is capable of generating such events. To improve the coverage of simulation, the design should be simulated with more expert experiences, and so configuring such constrained corner cases supported by configurable formal specification model to determine biases is extremely fruitful; most of the holes can be covered with these biasing.

8 EmGen: An Automatic Test-Program Generation Tool for Embedded IP Cores Conclusions In this paper, we describe EmGen, an automatic test-program generation tool designed for configurable embedded microprocessor cores. EmGen s modeling platform includes an configurable formal specification, which can describe the requirement of microprocessor configuration and verification plan completely and correctly, a heuristic test program generator with DBC and ST algorithms, which enable the generation of test programs across the full spectrum, from completely random through testingknowledge biased random, to highly directed, a reference instruction set simulator, which can provide reference results for all test cases, and a validation environment, which check the equivalence of simulations and reports the bugs automatically. Em- Gen has been taken into practice for the verification of a 32-bit configurable embedded microprocessor core. Experiment results have proven its flexibility, applicability and good performance. References 1. D. Campenhout, T. Mudge, J. Hayes, High-Level Test Generation for Design Verification of Pipelined Microprocessors, In proceeding of the 36th ACM/IEEE Design Automation Conference (DAC), (1999) S.Fine, A. Ziv, Coverage Directed Test Generation for Functional Verification Using Bayesian Networks, In proceeding of the 40th ACM/IEEE Design Automation Conference (DAC), (2003) R. Emek, et al. X-Gen: A Random Test-Case Generator for Systems and Socs, IEEE International High Level Design Validation and Test Workshop, Cannes(2002) 4. O. Lachish, E. Marcus, et al. Hole Analysis for Functional Coverage Data, In proceeding of the 39th ACM/IEEE Design Automation Conference (DAC), (2002) M. Bartley, D.Galpin, T.Blackmore. A Comparison of Three Verification Techniques: Directed Testing, Pseudo-Random Testing and Property Checking, In proceedings of the 39th ACM/IEEE Design Automation Conference (DAC), (2002) Haihua Shen, et al. Adaptive Test Program Generation for Embedded Microprocessor Core, In proceedings of the 1 st International Conference on Embedded Software and System (ICESS), (2004) Synopsys, Inc. Constrained-Random Test Generation and Functional Coverage with Vera, (2003) 8. Verisity Design, Inc. Specman Elite, (2004) 9. Jim Lipman, Configurable SoCs Give You Options, (2000)