Reliable Verification Using Symbolic Simulation with Scalar Values

Transcription

1 Reliable Verification Using Symbolic Simulation with Scalar Values Chris Wilson Computer Systems Laboratory Stanford University Stanford, CA, David L. Dill Computer Systems Laboratory Stanford University Stanford, CA, ABSTRACT This paper presents an algorithm for hardware verification that uses simulation and satisfiability checking techniques to determine the correctness of a symbolic test case on a circuit. The goal is to have coverage greater than that of random testing, but with the ease of use and predictability of directed testing. The user uses symbolic variables in simple directed tests to increase the input space that is explored. The algorithm, which is called quasi-symbolic simulation, simulates these tests using only scalar (0,1,X) values internally causing potentially conservative values to be generated at the outputs. Divide and conquer of the symbolic input space is used to resolve this conservativeness. In the best case, this method is as efficient as symbolic simulation using BDDs and, in the worst case, gives coverage and predictability at least as good as directed testing. 1. INTRODUCTION The ability to verify hardware designs has not scaled with the increase in design size over time. Although a lot of new methods have been proposed, they have not replaced directed and random simulation in most designs as the primary verification method. One reason for this is that new methods concentrate on improving efficiency with less consideration put on meeting the reliability requirements of mainstream verification. Reliability is the ability to produce at least some results, including predictable coverage and useful feedback, when memory or time limits are exceeded. Efficiency is the number of input patterns that can be verified with a given cost. We define cost to include manpower effort, computer resource usage, and project schedule time allocated. The basic methods used in verification can be characterized by their reliability and efficiency. First, directed tests are reliable because This work is supported by the MARCO/DARPA Gigascale Silicon Research Center (GSRC). We also thank Fujitsu Systems Technologies (FJST) for the use of their resources and designs. Permission to make digital/hardcopy of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. DAC 2000, Los Angeles, California (c) 2000 ACM /00/0006..$5.00 they have predictable coverage and do not run into any resource limitations, but they are not particularly efficient since each covers only a small part of the design. A single long random test may have the same coverage as a number of short directed tests making it more efficient, but random tests are less reliable because they are more difficult to debug and determine coverage for. Emulation increases efficiency even more and is proportionally more difficult to debug and determine coverage for. Model checking can give orders of magnitude more efficiency and is also considered to be very unreliable. Because directed and random testing are extensively used in mainstream verification while emulation and model checking are not, we conclude that reliability is an important requirement in mainstream verification. The goal is to improve mainstream verification methods. Our idea is to use symbolic simulation to gain efficiency, but with algorithms that degrade gracefully when resource limits are encountered. The basic idea is to use a scalar simulator that produces potentially conservative results coupled with divide and conquer of the symbolic input space to resolve any conservativeness. This results in efficiency comparable to random testing and predictability comparable to directed testing. Additionally, memory usage is a function of design size only, allowing this method to scale to designs beyond one million gates. Another key requirement in mainstream verification is the ability to support reactive tests. Reactivity is the ability of a test bench to change its behavior in response to how the circuit under test behaves. Quasi-symbolic simulation has been extended to support reactivity efficiently. 1.1 Symbolic Simulation Simulation consists of applying an input pattern to a circuit, simulating the circuit to determine the resulting output values, and then checking that the output values are as expected. This is written as the relation where and represent the set of circuit inputs and outputs respectively, and represents the set of values that are applied to the circuit and checked at the outputs. The antecedent( ) specifies what values are set on what inputs and the consequent( ) specifies the value expected on each output. The antecedent and consequent together form the environment for the circuit. We use the same set of values,, on the input and output side to indicate that the values put on the inputs, or a function of these values, are what we expect to appear at the outputs. For example, if we load a data value into a FIFO buffer, we expect this same value to appear at the output at some point.

2 Scalar simulation is simulation in which values,, range over the domain. Binary simulation is the subset of scalar simulation which ranges over the domain,. Verification consists of defining the total input space that needs to be simulated to ensure the design is correct. The input space is defined such that a single binary test covers one point of the input space. The total input space that needs to be verified is usually too large to cover completely, so directed and random testing work by sampling the input space. Symbolic simulation verifies a set of scalar tests in the input space with a single symbolic test. Since each scalar test may require a different value on each input, symbolic functions encoded using BDDs are used to represent all the possible values on each input. This is the method used in Symbolic Trajectory Evaluation (STE) [8]. The advantage of STE is that it can cover large input spaces efficiently and completely and that it is design size independent allowing scaling to large designs. However, the use of BDDs means that it is not reliable due to the well known BDD blow up problem [3]. When BDD blow up occurs, no coverage is obtained since the simulation does not finish and these cases can be hard to debug. 1.2 Quasi-Symbolic Simulation In quasi-symbolic simulation, values are not allowed to take on arbitrary functions and, consequently, BDDs are not used. Instead, the scalar domain is extended to include multiple, unique unknowns called symbolic variables. Test cases apply values from this domain to the simulator to produce output values to be checked. The algorithm consists of two parts. First, after input values are applied, a simulation engine quickly computes a potentially conservative scalar value at each internal node. Second, a decision procedure looks at the result of each run and divides the input space to resolve any unwanted conservativeness; multiple runs are performed to cover the entire input space. The rationale behind this is that, in large circuits, most internal node values do not propagate to outputs that are checked and so it is wasteful to generate exact values for these nodes. When the circuit is re-evaluated, only those nodes that actually propagate to checked outputs need be computed more exactly. The key to making this effective is to search the input space efficiently. This paper describes a search method that uses case splitting and depth-first search to get the desired reliability and efficiency. Outline We first describe the lattice that forms the basis of our abstraction method and then the basic simulation algorithm and search procedure. We then describe how this procedure is extended to handle reactivity and report on some experiments done to test our methodology. 2. BOOLEAN FORMULAS A circuit composed with a simulation relation creates a boolean formula that the simulator tries to satisfy. A boolean formula consists of strings over variables and the connectives,. Formulas are evaluated over the lattice where and represent 1 a unique symbolic variable and its complement having the variable identifier,. The lattice is defined as,, and. For soundness, the boolean connectives are required to be monotonic; that is, the relation must hold. Table 1 shows a monotonic the reason for this notation will become clear in Section 3.1 X Table 1: Truth Table for AND implementation of the AND function over two unique variables, and. 3. QUASI-SYMBOLIC SIMULATION A symbolic test case does two things. First, it creates an arbitrary number,, of symbolic variables, which along with the ternary values from the quasi-symbolic sub-domain that will be used in the test case. Secondly, it specifies what values from the sub-domain are put on each input in each time frame and a relation over any outputs that must be checked in each time frame. The simulator reads in the design and starts simulating from an initial state at time frame zero. It applies inputs specified by the test case for the current time frame and evaluates the circuit. It then evaluates any output relations for the current time frame, advances to the next time frame and repeats these steps until the test stops. Values of all output relations are ANDed and the result inverted to produce a final error output. If this result is then a bug is found and the test is said to be satisfiable. A indicates no bug and the test is unsatisfiable. If the result is then a procedure is invoked which examines this result to decide what to do. 3.1 Decision Procedure We want to choose a decision procedure that is both reliable and efficient. Sequential ATPG [4] is a search procedure that has similarities to symbolic simulation in that it is trying to produce a particular output value on a sequential circuit. Sequential ATPG algorithms work by either simulating forward from an initial state to the satisfying output, or proceeding from the output and justifying backwards to an initial state. Simulation always proceeds forward from an initial state. This can be inefficient in comparison to backwards justification, but backwards justification may timeout never finding a justifying state assignment, which does not meet our reliability goal. Search based methods have also been proposed for model checking. Boppana et. al. [2] describe a model checking procedure which converts the problem into a sequential ATPG problem. The robustness of their method, then, is determined by the robustness of the sequential ATPG program they use. Biere et. al. [1] use a similar method to translate the model checking problem into a combinational SAT problem, which they feed to a SAT solver. However, this method requires the state relation to be explicitly unrolled in the formula to be checked. Consequently, this method is not scalable to large designs, and its reliability is unknown. Ganai, Aziz, and Kuehlman [7] use search methods to exercise internal nodes that have not been exercised by random tests. It is possible to implement this algorithm within our framework, however we are concentrating on complete methods in this paper and so don t need to search over internal signal values.

3 Quasi-symbolic simulation evaluates the simulation relation over the circuit directly rather than by conversion to clausal form. A variant of the Davis-Putnam (DP) [6] procedure is used to search over the symbolic input variables. Forward simulation combined with searching over input variables means that leaf nodes of the search give some amount of input space coverage. DP uses depthfirst searching which gets to leaf nodes quickly. If time out occurs, it is likely to have reached at least some number of leaf nodes, and the longer the timeout, the more leaf nodes that have been reached. This gives us the reliability we are looking for. The basic DP algorithm is as follows: 1. Determine satisfiability by simulating the circuit. 2. If satisfiable, terminate with an error. If unsatisfiable, backtrack to a previous decision. Otherwise 3. Perform unit propagation of the symbolic variables if allowed. 4. Select a symbolic variable,, from the sub-domain,, to case split on. 5. Recursively do these steps with and. Unit propagation is an implication procedure that reduces the space that must be searched. For example, if a formula has the form,, where is a variable, then must be set to true to satisfy, eliminating the branch in the search tree. DP allows unit propagation to occur at any time, however, we will show later that our method must disallow unit propagation at certain times in order for the method to be complete. This is the only difference between our method and classical DP. The simulator uses a non-clausal unit propagation algorithm due to Dalal [5], called PCP (Propositional Constraint Propagation.) In Dalal s method, each sub-formula is evaluated in a bottom up manner and two sets, called the C-set and D-set, are associated with each sub-formula. If a sub-formula can be represented as a conjunction of literals 2 with some sub-formula,, then it has the C-set,. D-sets are constructed similarly from variables that are disjuncted with the sub-formula. Once the formula has been evaluated, if the D-set is non-empty, the formula is known immediately to be satisfiable. If the C-set is non-empty, all literals in the C-set are unit propagated to be true. The value of a sub-formula is denoted as if is the set of literals in the D-set or if is the set of literals in the C-set. A literal,, in this context has both a C-set and D-set exactly equal to the literal and is equivalent to our notion of symbolic variables since the value of this subformula is exactly or when the literal is or respectively. 3.2 Variable Selection Heuristic The algorithm uses a heuristic to select the best variable to split that minimizes the size of the search tree. The heuristic starts by associating a variable with each internal signal in the circuit. This variable is selected from the inputs to the function driving the signal as follows: A literal is a variable or its negation. 1. At a primary input, if the input is a literal, the associated variable is the literal variable. 2. If the C-set or D-set of the output is non-empty, propagate one of the variables in these sets. 3. If one of the inputs is a controlling value, then propagate that input s associated variable. 4. Otherwise, select the associated variable of the input that has the lowest index. The associated variable that appears at the final output is the one that is case split on. This algorithm works well because it only selects variables that can affect the consequent, allowing us to have symbolic variables on don t care inputs. Don t care inputs are inputs that should not affect any checked outputs. For the circuit to work correctly, there must be some gate in the circuit which blocks each don t care value due to a controlling value at the other input. Consequently, if symbolic variables are labeled care or don t care based on which primary input they were put on, then don t care variables will not be propagated to checked outputs in a correctly working circuit. This allows large input spaces to be easily searched since we can introduce large numbers of don t care symbolic variables with no fear of blow up. The test cases used in our experiments routinely have over 1000 symbolic variables, most of which are don t cares, with the actual number of variables in the search tree usually being less than Soundness Soundness of a symbolic simulator is the property that when a symbolic test is proved unsatisfiable (has no bugs,) all scalar tests represented are also unsatisfiable. Since the only operation the divide and conquer algorithm performs is the substitution of constants for symbolic variables, soundness is a direct result of monotonicity of the boolean operators. 4. REACTIVITY Reactivity allows specifying time varying behavior in a time invariant way and is important in mainstream verification. A reactive test may, for example, assert some signal and then wait for the circuit to respond before continuing the test. Reactivity is difficult to implement in a symbolic simulator because different scalar input patterns represented by the symbolic input pattern may have to wait different amounts of time before continuing. A test consists of a set of threads that execute concurrently to implement the antecedent and consequent. Threads can set signal values to quasi-symbolic values, check the correctness of outputs, suspend thread execution for a specified delay, globally stop the test, and make thread execution conditional upon a guard expression. Reactivity is implemented by adding a wait statement within a thread which specifies an expression and delay value. When the wait statement is executed, the expression is evaluated, and if true, execution continues. If false, thread execution is suspended for the amount specified by the delay, after which, the wait statement is re-executed. It is possible that the wait expression will evaluate to the value, meaning that it may be true and false at the same time. To maintain monotonicity, the simulator must both continue execution of the thread and continue waiting at the same time. This is handled by

4 creating a copy of the thread, called a virtual thread which starts executing at the statement after the wait and is guarded with the wait condition being true. The original thread continues to wait, guarded with the wait expression being false. As long as the wait expression evaluates to, the simulator forks off new virtual threads and continue to wait in the original thread. The semantics of stopping a test are that if any thread executes a stop statement, the entire test is stopped. Again, to maintain monotonicity, if the thread guard value is when a stop statement is encountered, then the test must both stop and continue executing. For correctness, it is necessary to stop the simulation only when a stop statement is encountered. Since the guard condition for a stop could also be, the decision procedure needs to prove that the test really stops at this point by case splitting on this guard condition before case splitting on any error conditions. Unit propagation cannot be used in this case since this is incomplete. The problem is that the stop guard condition case splitting is trying to prove that the simulation can stop in this particular cycle, not whether it may stop in this or some future cycle. Since the test case should stop eventually for all case split values, by not allowing unit propagation, the entire search tree will be enumerated, allowing the simulator to reach all cycles in which the test can possibly stop. To illustrate this, consider the following example. A four bit counter is initially loaded with a value and then decremented. The counter outputs a signal when the count reaches zero. For any binary vector loaded into the counter, the output will always eventually be asserted. Thus, if a test consists of loading the counter, waiting for the output to be asserted, and then stopping, the test case will always eventually terminate. Now consider a symbolic vector representing the set of all possible count values being loaded into the counter. The initial state and output after simulating one cycle will be as follows: Since the stop condition is non-zero, the simulator stops at this point and case splits. If unit propagation was allowed, all variables would be set to 0 as indicated by the stop condition C-set. In the next evaluation in the first cycle which means the test case has been proved to actually stop in the first cycle. The decision procedure now backtracks, but since all variables were unit propagated, the search would terminate immediately and would have only tested the case in which the test stops after one cycle. But, clearly, the test can stop at all lengths up to the maximum counter value. Completeness Reactive tests are inherently incomplete, even when limited to the binary domain. For example, a wait statement may wait forever for a signal to be asserted due to a bug in either the circuit or test case. This extends to the symbolic domain since symbolic tests represent sets of binary tests. What we are interested in showing is that if all binary tests represented by a symbolic test are complete, then the symbolic test is complete. By disabling unit propagation in the stop condition case splitting, we correctly explore all test case lengths. The search will also terminate since in any leaf node, a scalar stop value will be produced, and in any abstract node, an value will be produced for the stop condition at a valid stop point. The search is guaranteed to terminate because there are a finite number of symbolic variables. Therefore, the test will always terminate and be complete. 5. EXPERIMENTS We have implemented a Verilog based quasi-symbolic simulator that supports hierarchical, gate level models, is event-based, and is semantically equivalent to a scalar Verilog simulator. RTL simulation is supported by synthesizing hierarchical RTL descriptions into equivalent gate level models. We implemented our own test case language because Verilog does not support the creation of symbolic variables and the RTL subset does not have some of the language constructs needed for test benches. We have performed two experiments, the first measures the efficiency of our method in finding a bug and the second demonstrates its reliability. The simulator runs 3 at a speed of 85K events/second and four time frames/second for the circuit we used. Test lengths ranged from around 30 to 100 time frames. The simulator plus design required 36MB of main memory and this value did not grow significantly as a test ran. 5.1 Test Design Description The design we are using is an industrial bus bridge chip called the MCU [9]. We are modeling the bus interface section of the chip which has approximately 140K gates and 2,402 state bits. This chip was taped-out, went through bringup and subsequent followon versions were designed. In the process of verifying these followon designs, bugs were found in the original design that were not caught in the original verification or in bringup. The bug we are targeting in our first experiment is in this category. 5.2 Experiment 1: Efficiency Our method improves efficiency compared to directed and random testing because of its completeness by allowing the user to rule out large parts of the input space quickly when looking for bugs. However, most of the effort in verification is spent in writing and debugging test cases. We expect this to be no different for symbolic simulation and so we need to show that our method does not cause significant bottlenecks when bugs are found. In this experiment, we developed a test case to find a known, but not well characterized bug. We wrote an initial test case and then recorded the number of evaluations (nodes in the search tree) and execution time of each run as we tried to debug it. After performing a number of runs, it became apparent that there were basically four possible results: 1. (SAT) A test case protocol violation in driving the requests onto the bus would occur. These typically were found with a few evaluations and required no backtracking. 2. (TIMEOUT) The test would timeout waiting for a response from the circuit. These were also typically due to test case protocol violations. On a Sun Ultra-5 workstation with 128MB of memory running SunOS 5.6.

5 case no. evals time(sec) rand evals SAT TIMEOUT LEAK UNSAT8j UNSAT8k UNSAT UNSAT UNSAT UNSAT UNSAT UNSAT UNSAT Table 2: Test Case Development Performance Figure 1: Size of largest completed subtree vs. maximum search tree size 3. (LEAK) Don t cares would leak through to a checked output and the simulator would have to case split on the don t cares to determine satisfiability. Usually, they did not just leak through, they flooded through resulting in a long chain of case splitting. These are the cases that detected the real bug in the hardware. 4. (UNSAT) The test case would not detect a bug because we were issuing the wrong request type. This would result in the simulator saying the test is unsatisfiable and usually required a small number of case splits. A total of 51 test cases were created before the design bug was discovered, requiring 10 working days of effort from start to finish. Table 2 summarizes the results for each test case written. For the SAT and TIMEOUT cases, the number of evaluations and execution time is the average over these cases, with the number for each given in the column labeled no. There were nine UNSAT cases, each of which is listed separately. Each test was also run using randomly generated ones and zeroes in place of symbolic variables with the average number of evaluations required to find the bug for these cases reported in the column labeled rand evals. Each of these cases is compared to random simulation. In the cases with test case errors (SAT and TIMEOUT,) both quasi-symbolic and random simulation caught the test case errors quickly with random simulation being faster. However, neither method was a bottleneck since the time to dump a waveform trace for viewing was significantly longer than the simulation time in both cases. The leak through case that actually caught the bug consists of a single long series of case splits with no backtracking. This was due to a large fan-in ECC circuit driven by don t care inputs which were set to symbolic variables in the test case. Random simulation was faster in this case, however the quasi-symbolic simulator prints out what variables are being split on and it was immediately obvious that it was splitting on don t care variables. It was straightforward to then eliminate most of the don t cares to reduce the case splitting to a handful that finished quickly. The UNSAT cases were due to the uncertainty in where to look for the bug. These cases cannot be compared with random simulation since random simulation is incomplete. Random simulation can give a hint that these cases may be unsatisfiable, however it was a real advantage to know that we needed to look elsewhere for the bug. Quasi-symbolic simulation was clearly better for these cases. In summary, for the cases that found test case bugs, quasi-symbolic simulation did not cause a bottleneck in finding the design error. For those those cases that did not detect bugs its completeness gave it an advantage over random simulation. 5.3 Experiment 2: Reliability For our second experiment, we chose to verify the bus to network data transfer logic. This test varies the number of requests, request type, and timings between various phases of the requests. There were no known bugs in the area being testing, so we expected the result of this test to be unsatisfiable. To measure reliability, we performed a number of runs in which we increased the search space in each run. We then counted the number of evaluations that must be performed to prove the test case unsatisfiable with a timeout limit of 100 evaluations. If the test case exceeds the evaluation limit, we measured the largest completed subtree in the search. The largest completed subtree is the highest node in the tree (the root node is the highest) for which all leaf nodes have been evaluated. The higher the node, the more input space that has been covered. Figure 1 shows the results of running this test when the number of symbolic decision variables is scaled from 11 to 26. Decision variables are those that are expected to appear at checked outputs; that is, they are not don t care variables. The X-axis counts the number of symbolic decision variables in the complete search tree and the Y-axis gives the number of symbolic decision variables in the largest subtree that was completed in a run. The dotted line in the plot indicates the case when the largest explored subtree is the entire tree. The plot shows that for the time limit imposed, our method was able to prove unsatisfiability for a maximum tree size of 16. Beyond that, the largest subtree explored was a subset of the entire tree, but as the tree grew, the covered subset was at least as large as the complete tree for the cases that did complete, except at the transition point. Thus, even though these cases hit a hard resource limit, our method gave coverage that was at least as good as a smaller test that did not time out. A disadvantage of quasi-symbolic simulation compared to random simulation is that when the highest completed node is below the root node, the symbolic variables case split above the highest completed node have only been set to one value for all evaluations. A

6 random simulator would, in all likelihood, set all variables to both and at some point during a test. However, this problem is easily overcome by re-running the test with a different variable order if it times out. 5.4 Conclusions We have stated that reliability is an important important factor in mainstream verification. Our goal is to improve mainstream methods and our strategy is to ensure reliability first, and then work on improving efficiency. Symbolic simulation can efficiently search large input spaces and its completeness allows ruling out large input spaces, which is a significant advantage over random simulation when looking for bugs. It can also use short tests to give the same coverage as long random tests, making tests easier to debug. However, traditional symbolic simulation based on BDDs does not meet our reliability criteria. Quasi-symbolic simulation uses algorithms that use scalar values internally and satisfiability checking methods to search over the set of symbolic test cases allowing it to degrade gracefully in the face of resource limits. The combination of predictability and simple test case debugging make our method comparable to directed testing in its ease of use. [8] C.-J. Seger and R. E. Bryant. Formal verification by symbolic evaluation of partially-ordered trajectories. Formal Methods in System Design, 6(2): , [9] W.-D. Weber et al. The mercury interconnect architecture: A cost-effective infrastructure for high-performance servers. In Proc. of the 24th Annual Intl. Symp. on Computer Architecture (ISCA97), In our experiments, we showed that our method had better efficiency than random testing in ruling out input spaces to search for bugs and that it does not introduce significant bottlenecks when bugs are encountered. We also showed that our method does degrade gracefully when resource limits are encountered. In summary, quasi-symbolic simulation has the ease of use and predictability of directed testing and the efficiency of random testing while meeting the reliability goals of mainstream verification. Thus, we believe that quasi-symbolic simulation can be used as one of the primary methods of finding the majority of bugs in large designs with the advantage that it can perform verification tasks not possible with either random or directed testing. 6. REFERENCES [1] A. Biere et al. Verifying safety properties of a powerpc microprocessor using symbolic model checking without bdds. In Proceedings, Computer Aided Verification (CAV 99), LNCS 1633, pages 60 71, [2] V. Boppana et al. Model checking based on sequential atpg. In Proceedings, Computer Aided Verification (CAV 99), LNCS 1633, pages , [3] R. E. Bryant. Graph-based algorithms for boolean function manipulation. IEEE Transactions on Computers, C-35(8): , August [4] K.-T. Cheng. Gate-level test generation for sequential circuits. ACM Trans. Design Automation of Electronic Systems, 1(4): , October [5] M. Dalal. Efficient propositional constraint propagation. In Proc. of the Tenth National Conf. on Artificial Intelligence (AAAI-92), pages , [6] M. Davis, G. Logemann, and D. Loveland. Machine program for theorem-proving. Communications of the ACM, 5(7): , [7] M. Ganai, A. Aziz, and A. Kuehlman. Augmenting simulation with symbolic algorithms. In Proc. of 36th Design Automation Conf., pages , 1999.