Using VCS with White-Box Verification Techniques

Transcription

1 Thomas L. Anderson 0-In Design Automation, Inc. ABSTRACT Because of its simulation efficiency, VCS is an excellent tool for traditional "black-box" verification, in which a testbench stimulates inputs and checks outputs for expected results based upon the design specification. With the large size of today's ASIC and IC designs, it is becoming more difficult to achieve controllability and observability of all parts of the design from primary inputs and outputs. The risk is that bugs deep inside the design will be missed because their effects were never observed when running the testbench. One solution is the use of "white-box" techniques to supplement traditional specification-based black-box simulation. These techniques include formal verification of design properties, directed tests that are designed to trigger certain internal conditions, embedded checkers that monitor for specific behaviors deep inside the design and formal verification targeting the embedded checkers. Such techniques are especially valuable with VCS since any overhead from the additional logic has minimal effect upon the simulation speed. The white-box approach is especially beneficial in a design environment with reused IP blocks. It can be very hard to reuse an IP testbench once the IP block is included in a larger chip. Embedded checkers can verify IP blocks both in standalone VCS simulations and in chip-level VCS simulations after IP integration, allowing white-box verification techniques to be leveraged by both IP providers and IP users in a reuse-oriented design.

2 1.0 Introduction As large chip designs get more and more complex, functional verification is becoming the dominant task in the development process. A widely cited 1998 case study from Nortel [2] reported that verification consumed two-thirds of a product design cycle. Many other companies have confirmed similar experiences, with anecdotal accounts of functional verification occupying as much as 90% of some projects. Reuse-oriented designs, in which major blocks are recycled from existing designs or acquired from external IP suppliers, tend to shift the balance of effort away from design and toward verification. Under this pressure, development teams are increasingly looking for new and innovative ways to verify designs more quickly. 2.0 Traditional Black-Box Verification Most complex chip designs start with a paper specification. Typically, this specification includes a high-level description of functionality and definitions for all chip inputs, chip outputs and architecturally-visible state elements. This specification is transformed into RTL, gates or transistors by designers and the resulting implementation must be verified to conform to the original specification. This verification effort is typically the responsibility of dedicated verification engineers who work alongside the hardware designers. The idea of black-box verification is fundamental to this method of operation. It is taken as a given that a designer should not write the verification tests for his or her own logic, since the same assumptions will be made in the tests as in the design. If a verification engineer goes back to the original specification and writes tests using his or her own interpretation, this provides crosschecking between the design implementation and the verification test. Further, many verification engineers argue that they should know as little as possible about the details of the implementation so that the design is verified strictly from the specification. Verifying a design exclusively by writing tests to manipulate chip inputs and monitor chip outputs is the essence of the black-box approach. In theory, black-box testing should be sufficient to verify that a design implementation matches the specification and is therefore functionally correct. However, there are several issues: The specification may not be complete, especially in terms of corner-case behavior. It may be very hard to set up all specified operations strictly from the chip inputs. It may be very hard to check all specified behavior strictly from the chip outputs. It may be very hard to track down the source of bugs exhibited at the chip outputs. Certain types of implementation errors are hard to stimulate with functional tests. The last issue may be the most difficult for many complex chip projects. Consider the case of a fairly common design bug in which a FIFO is written even after it is full. The effect of such a FIFO over-run is likely to be corrupted data, but unless a functional test happens to trigger a FIFO full condition and the associated overwrite, the bug will remain undetected. A FIFO is generally not an architecturally visible element, and so a verification engineer following a strict 2

3 black-box approach will not know about the FIFO and will not know (or care) whether or not the full and empty conditions are triggered by the functional test suite. Even if a functional test happens to trigger the FIFO over-run, other conditions within the chip during the test may prevent the corrupted data from propagating to the chip outputs. Even if the corrupted data does propagate to the outputs, it will not be detected unless the verification engineer has a test monitor set up to detect the problem. Finally, even if the verification test detects the corrupted data, it may be very hard to track back to the source of the problem deep within the chip. Especially in networking and telecommunications applications, data corrupted in an internal FIFO may take thousands of cycles to propagate to the chip outputs. 3.0 Some Common Extensions The limitations of black-box verification for large, complex chips suggest that supplemental approaches are needed. Increasingly, development teams have concluded that the verification engineers cannot rely exclusively on black-box methodology when writing tests. The typical project flow requires the verification engineers to first write the black-box tests strictly from the specification, and then to write additional tests designed expressly to trigger certain interesting internal behavior such as the FIFO full and empty conditions mentioned earlier. Such a process may not require the verification engineer to become an expert on the implementation, but it does mean that he or she needs to work with the design engineer to understand the worry cases in the design. Most of the limitations of black-box verification still apply. Even if a sufficiently important worry case is defined, it may be hard to write a test that triggers the condition from the chip inputs. It also may be hard to observe bug effects at the chip outputs, which leads verification engineers to probe and display some internal chip signals in VCS simulation to try to locate bugs. Properly hooking up a monitor to internal signals and keeping the monitor updated as the design changes can be significant challenges. The main issue with this approach is that it is only as good as the tests written by the verification engineer. It is hard to think of all the interesting internal behavior to be exercised and hard to verify that the tests do what is intended to trigger this behavior. Verification engineers lose sleep over what cases they may have forgotten when writing tests. The usual way to try to close the gap for complex designs is to supplement the hand-written tests with some form of extended simulation that stimulates the design with random vectors or random sequences. Developing a testbench that stimulates the design in new legal ways as opposed to possibly illegal random patterns on all inputs is hard. The specification for the chip should define the legal input combinations and sequences, but these need to be captured in the test generation code of the extended simulation testbench. Further, the chances of a random test actually triggering a bug deep inside the design and the chances that the effects will randomly propagate to the outputs and be detected there are distressingly small. The only way to make significant advancement beyond black-box techniques is to improve the controllability or the observability of the design during the verification process. The latter is 3

4 easier; many design and verification engineers use Verilog $display statements to detect interesting conditions either error cases or specific deep behavior to raise alerts during simulation when these conditions occur. For example, a designer might choose to display a warning message whenever an FSM transitions to an error state. Consider the previous case of a FIFO that can over-run and corrupt data. If the FIFO has a full signal available, then it is easy for a design or verification engineer to display a message if the enqueue signal is asserted when the FIFO is already full. If the FIFO corruption is detected when it happens, rather than when the effects eventually propagate to the chip outputs many cycles later, it can be much easier to track down the source of the bug. Of course, if signals are not conveniently available, then the engineer may have to add special logic just to support the check and display. Such logic must be surrounded by Synopsys synthesis off/on directives to ensure that it does not get fabricated in the chip, and special treatment may also be needed to ensure that the check logic does not affect the results from RTL code coverage analysis tools. Improving simulation controllability is a harder problem, and solutions tend to fall into one of two domains. The first approach is testbench automation, which is a direct descendent of ATPG tools for improving the fault coverage of manufacturing test vectors. The argument is that if ATPG can improve fault coverage, then testbench automation can improve RTL code coverage in much the same way. This argument misses two key points. The first problem is that there is no bug model for RTL that corresponds to the stuck-at fault model for testing chips in manufacturing. Even if testbench automation greatly increases the RTL code coverage and exercises the design in a way that stimulates a bug, detection of the bug relies on detection of bug effects at the chip outputs, or perhaps by some sort of internal $display check. The second problem is that virtually all ATPG algorithms rely on the ability to freely manipulate the design inputs to set up conditions to trigger faults. Since testbench automation operates in the functional verification domain, it cannot make the same assumption. All input changes and sequences must conform to the specification, since a generated test is worthless if it triggers a bug via illegal inputs. This means that the engineer using the tool must describe a set of constraints for legal input behavior, a task that most users find extremely difficult. 4.0 The Role of Formal Verification Such techniques as $display statements are the first steps toward what is sometimes called whitebox verification. This term stresses the contrast with black-box verification, since knowledge of the chip design is used during verification, rather than being intentionally ignored. One wellknown white-box approach is formal verification, in which mathematical techniques are used to prove an assertion or property of the design. The property to be proven may be related to the chip s overall functional specification, or may represent internal design behavior. The FIFO example described earlier is an interesting case for formal verification. A statement such as the FIFO will never enqueue when it is already full makes a logical target for a formal proof. If a formal verification tool can prove mathematically that this assertion is always true, i.e., that there is no way that the design can ever over-run the FIFO, then this is a very valuable piece 4

5 of knowledge. This knowledge relieves the design and verification engineers from having to develop many test cases or random sequences to stress-test the design to see if a FIFO error ever occurs. The goal of formal verification is wonderful, and formal verification does prove useful in some circumstances, but there are some serious limitations and barriers to its wide adoption. Users sometimes consider using formal verification to replace functional tests and simulation as a way to verify designs. One problem is that it is prohibitively difficult to write a set of properties or assertions that completely represents all the intended behavior of a design. The result is that formal verification has found its widest use as a supplement to simulation rather than as a replacement. A verification engineer may write only a few key properties and use a formal tool to try to prove them in parallel with more traditional simulation tests. In the course of proving a property, a formal algorithm essentially explores all the possible ways to manipulate a design. All input changes must conform to the constraints for legal behavior but, as noted previously, specifying these constraints is a challenge. In extreme cases, such as a serial bus controller or a deeply embedded processor, input sequences many thousands of cycles long may be required to uncover the one case that disproves an assertion. The constraints must be complete to ensure that this entire sequence is within specification. Because it must explore a design exhaustively, formal verification can run only on designs of limited size. Conventional wisdom dictates that formal techniques are best applied on design blocks of less than 10,000 gates. Thus, complete formal verification of a contemporary multimillion gate chip requires runs on hundreds of small blocks. This only exacerbates the constraint problem. Few designers fully specify interfaces between blocks as small as 10,000 gates and so constraining the inputs of blocks this small may require a great deal of new specification work to define allowed behavior on these interfaces. The difficulty of writing enough properties to provide significant verification and the difficulty of developing sufficiently robust constraints are significant barriers to the use of formal verification. As a consequence, formal tools are used almost exclusively by specialists. This is in sharp contrast to VCS-based functional testing, which enjoys a large, trained user base. 5.0 Introduction to Semi-Formal Verification It would be highly desirable to be able to take advantage of the power of formal verification while easing some of its restrictions and benefiting from the huge knowledge base for VCS-based simulation techniques. A new approach, sometimes dubbed semi-formal verification, is emerging to meet this need. Four key attributes of semi-formal verification differentiate it from more traditional techniques: Formal methods are used to amplify existing simulation tests. Embedded checks define the properties targeted by formal methods. Embedded checks define the input constraints. Semi-formal verification supplements, not replaces, simulation. 5

6 The key to this approach is viewing semi-formal verification as amplifying the design and verification engineering knowledge already embodied in a testbench and functional test suite. The rationale is that the power of the formal algorithms should be used to explore new behaviors in the design rather than to duplicate the stimulus already available from the existing black-box tests. During any VCS simulation run, seed states are captured to serve as starting points for the formal methods. A semi-formal verification tool starts from each state in the seed, and tries to find new stimulus sequences that will violate assertions about the design. In so doing, the tool discovers legal behaviors that could have occurred but did not in the functional tests. The assertions targeted may include everything from simple formal verification properties to complex rules about how interfaces and internal data structures must operate in the design. Some of these complex rules can be hard to specify within the limitations of property languages, so a better approach is to use embedded checkers to specify these rules. Checkers are monitors that the design or verification engineer can add to the design to specify rules of operation. Ideally, the monitors can be expressed directly in Verilog so that the engineer does not need to learn a new language. Embedded checkers might include: checkers for data path elements, such as verifying that data is not lost while flowing down a pipeline or corrupted in a FIFO or memory checkers for control structures, such as verifying that an arbitration scheme is correct or that a state machine is operating properly checkers for interfaces, such as verifying multi-cycle timing relationships between signals or checking that a tri-state bus is always driven Complex checkers that directly reference buses and design structures are more intuitive for the designer and provide interesting, meaningful targets for the formal algorithms. If a semi-formal tool discovers a novel stimulus sequence that can over-write data in a FIFO or violate an interface protocol rule, this is a bug that might not have been found with black-box techniques. Unless a verification engineer had written a test to set up the same conditions and sequence discovered by the formal methods or unless random tests happened to trigger this behavior the bug would have made it into silicon. The third category of embedded checkers, those that monitor for interface rules, define legal behavior across an interface. Therefore, a set of checkers on an interface can be converted trivially to constraints. This ensures that a semi-formal tool only uses legal manipulations on the inputs as it creates new stimulus sequences that deviate from the seed. Instrumenting an interface with a set of checkers to define the protocol is generally easier than using a constraint language to capture the rules for legal and illegal behavior. Checkers that specify semi-formal targets and constraints may also have value in the functional simulation process. If these checkers have simulation models, they can be included in any VCS simulation to detect whether any existing functional test violates a checker rule. If this is the case, then a design bug has been detected and it should be fixed before running semi-formal verification. As stated earlier, the goal is to reserve the use of formal methods for exploring new behaviors that were not already reached by existing tests. 6

7 Checkers may be included in functional simulation in one of several ways. They can be included directly in the RTL, although this requires Synopsys off/on directives to ensure that the checker logic is not synthesized into the chip. A better approach is to place the checkers into a separate file that uses Verilog hierarchical references to monitor the signals from the RTL. This file is then included in VCS runs but not in synthesis compilations. A hybrid approach is the generation of the separate checker file from directives in the RTL. If the directives use the Verilog comment syntax, they are ignored during synthesis and require no special handling. By leveraging existing tests and testbenches, semi-formal verification is an add-on to traditional VCS functional simulations. This approach is consistent with the prevailing view that formal verification is best used as a supplement to simulation rather than as a replacement. Further, some semi-formal tools may also leverage VCS to confirm checker violations. If the stimulus sequence discovered by the formal methods is run in a simulation, and the checker reports a violation, this verifies that the bug can be reproduced in the design or verification engineer s own environment. Once a VCS simulation run exhibits the violation, the engineer can use such familiar methods as dumpfiles and waveforms to track down the source of the bug. 6.0 Design Environments and Design Reuse One advantage of the semi-formal verification approach is that it is applicable throughout the development process. Once checkers are added to a block, they can be included in any VCS simulation at the block- or full-chip level. After simulation with checkers, tests for individual blocks can be amplified by formal methods to make the blocks more robust before they are integrated into the full chip. Block-level testing is minimal on many projects and amplification provides a way to improve the situation without requiring any new tests to be written. At the full-chip level, it makes sense to leave the checkers in the individual blocks since each new test may stimulate a particular block and a particular checker in new ways. These checkers can also help to pinpoint the cause of errors in full-chip simulations. Chip-level verification engineers are likely to supplement the intra-block checkers with additional checkers on the main interfaces between blocks and on any on-chip buses. Running chip-level functional simulations with the full set of checkers within and between key blocks can provide a significant increase in the observability of bugs deep within the design. Finally, it often makes sense to amplify seed states captured for key blocks from chip-level simulation, especially blocks that had no block-level testbenches. The inputs, outputs and internal state of a block may be captured from VCS in a manner similar to generating a dumpfile. This state may be used as a seed for formal methods to amplify the effects of the chip-level functional tests on that particular block. There is no fundamental difference between a seed captured from a block-level testbench and a seed captured from a particular block in a chip-level testbench. In either case, a semi-formal tool can use this seed to improve controllability and explore the block in new ways. The idea of using block-level checkers in full-chip simulations is especially valuable in a design reuse environment. If the block is designed for reuse, either within the same company or as a 7

8 commercial IP product, checkers help to document design intent and to ensure that the engineer integrating the block uses it properly. Specifically, a robust set of checkers on the interface between the IP block and the user s application logic is very valuable. If the application logic does not obey proper protocol rules on the interface, a checker immediately reports that the block is not being used properly. 7.0 White-Box Verification Results A recently published case study [1] described a development project that was carefully monitored to assess the impact of using the white-box techniques of embedded checkers and semi-formal verification. This project involved the design and verification of a Dual-CPU PCI Bridge, as shown below. PCI Bridge CPU CPU req gnt req gnt CPU Bus Arbiter req gnt ID Table ID Fo rm at CPU Interface ID Count Fo rm at PCI Master Memory PCI Bus Memory Interface PCI Slave M e m or Two examples of bugs found by white-box techniques in this design are especially illustrative of the approach. The first bug was found in simulation by one of the checkers attached to the PCI bus to monitor for correct protocol behavior. The PCI slave in the memory interface asserted the Trdy signal before the DevSel signal, a violation of the PCI protocol. This bug was detected when the checker reported the violation during a block-level test that stimulated the PCI slave, and turned out to be due to a slave state machine problem. The second bug example is a serious corner-case bug that almost certainly would not have been found by traditional techniques. The scenario is that one CPU issues a two-cycle write, but is interrupted in the middle when the arbiter gives the processor bus grant to the other CPU. The effect was that only one word of data is written into the FIFO connecting the CPU and the PCI master. However, the PCI master recognizes the command as a two-cycle write and therefore pops two words from the FIFO without checking validity. The PCI master designer had never considered the possibility that a two-cycle write could be interrupted on the CPU bus. This bug was detected by an embedded FIFO checker whose do not dequeue when empty rule was violated when the PCI master read the invalid second word. This rule was never violated in functional simulations, since no test was written for an interrupted two-cycle write. However, when semi-formal verification amplified the seeds from the functional tests, and used the FIFO checker rule as a target for its formal methods, it was able to discover and report the bug scenario. Thus, it found a serious bug that otherwise would have remained undetected. 8

9 8.0 Conclusion The complexity of today s large chip designs is causing a verification crisis. It is almost impossible for verification engineers to think of all corner case conditions and to write enough black-box tests. Even when a black-box test detects the effects of a bug at the chip outputs, it can be difficult and time-consuming to track down the bug source deep within the chip. Random tests and hardware-software co-verification provide additional stimulus but do not necessarily find more bugs. White-box verification techniques are increasingly being used to supplement traditional functional verification via tests in simulation. White-box methods, especially when they leverage the power of semi-formal verification, can find bugs more quickly, closer to the source, and discover corner-case bugs that would otherwise have made it into silicon. 9.0 References 1. Using Embedded Checkers to Solve Verification Challenges by Scott Switzer and David Landoll (published in DesignCon 2000 IP World Forum Proceedings) 2. Functional Verification of ASICs in Silicon Intensive Systems by Allan Silburt (published in DesignCon98 On-Chip System Design Conference Proceedings) 3. Formal Methods Technology Validation with Guided Search of the State Space by C. Han Yang and David L. Dill (published in Proceedings of the 35th Design Automation Conference) 4. Hybrid Verification Using Saturated Simulation by Adnan Aziz, Jim Kukula and Tom Shiple (published in Proceedings of the 35th Design Automation Conference) 5. A Decision Procedure for Bit-Vector Arithmetic by Clark W. Barrett, David L. Dill and Jeremy R. Levitt (published in Proceedings of the 35th Design Automation Conference) 6. Hardware/Software Co-Design of the Stanford FLASH Multiprocessor by Mark Heinrich, David Ofelt, Mark A. Horowitz and John Hennessy (published in Proceedings of the IEEE, Vol. 85, No. 3, March 1997) 7. Integrating Formal Verification Methods with a Conventional Project Design Flow by Th. Eiríksson (published in Proceedings of the 33rd Design Automation Conference) 8. Validation Coverage Analysis for Complex Digital Designs by Richard C. Ho and Mark A. Horowitz (published in ICCAD '96 IEEE/ACM Digest of Technical Papers) 9. Architecture Validation for Processors by Richard C. Ho, C. Han Yang, Mark A. Horowitz and David L. Dill (published in Proceedings of the 22nd Annual International Symposium on Computer Architecture) 9