ECE 587 Hardware/Software Co-Design Lecture 11 Verification I

Transcription

1 ECE 587 Hardware/Software Co-Design Spring /23 ECE 587 Hardware/Software Co-Design Lecture 11 Verification I Professor Jia Wang Department of Electrical and Computer Engineering Illinois Institute of Technology February 13, 2018

2 ECE 587 Hardware/Software Co-Design Spring /23 Reading Assignment This lecture: Next lecture: 7.3

3 ECE 587 Hardware/Software Co-Design Spring /23 Outline Verification Simulation Based Methods Equivalence Checking

4 ECE 587 Hardware/Software Co-Design Spring /23 Functional Verification Analysis and reasoning on a computer model of the system Before manufacturing Establish confidence in functional correctness before the product is shipped Critical for systems where safety is the first concern Prevent costly recall for non-critical systems as well Designers need to make sure the model at each step of design does reflect the original intent. Catch bugs as early as they are introduced so one can locate them effectively

5 ECE 587 Hardware/Software Co-Design Spring /23 Functional Verification Analysis and reasoning on a computer model of the system Before manufacturing Establish confidence in functional correctness before the product is shipped Critical for systems where safety is the first concern Prevent costly recall for non-critical systems as well Designers need to make sure the model at each step of design does reflect the original intent. Catch bugs as early as they are introduced so one can locate them effectively

6 ECE 587 Hardware/Software Co-Design Spring /23 Functional Verification Analysis and reasoning on a computer model of the system Before manufacturing Establish confidence in functional correctness before the product is shipped Critical for systems where safety is the first concern Prevent costly recall for non-critical systems as well Designers need to make sure the model at each step of design does reflect the original intent. Catch bugs as early as they are introduced so one can locate them effectively

7 ECE 587 Hardware/Software Co-Design Spring /23 Simulation Based Verification vs. Formal Verification Both verification methods need a golden reference as part of the specification. Simulation based methods: stimuli (inputs) and monitors (expected outputs) Formal methods: mathematical model of desired properties Simulation-based methods are effective to catch bugs at early design stages, though only as good as the stimuli. Formal methods can provide a proof of correctness, but require more effort at design time. Simulation is still predominant while formal methods are catching up. More formal verification tools become available. More designers are trained to use these tools. e.g. most gate-level verifications are based on formal methods nowadays

8 ECE 587 Hardware/Software Co-Design Spring /23 Simulation Based Verification vs. Formal Verification Both verification methods need a golden reference as part of the specification. Simulation based methods: stimuli (inputs) and monitors (expected outputs) Formal methods: mathematical model of desired properties Simulation-based methods are effective to catch bugs at early design stages, though only as good as the stimuli. Formal methods can provide a proof of correctness, but require more effort at design time. Simulation is still predominant while formal methods are catching up. More formal verification tools become available. More designers are trained to use these tools. e.g. most gate-level verifications are based on formal methods nowadays

9 ECE 587 Hardware/Software Co-Design Spring /23 Simulation Based Verification vs. Formal Verification Both verification methods need a golden reference as part of the specification. Simulation based methods: stimuli (inputs) and monitors (expected outputs) Formal methods: mathematical model of desired properties Simulation-based methods are effective to catch bugs at early design stages, though only as good as the stimuli. Formal methods can provide a proof of correctness, but require more effort at design time. Simulation is still predominant while formal methods are catching up. More formal verification tools become available. More designers are trained to use these tools. e.g. most gate-level verifications are based on formal methods nowadays

10 ECE 587 Hardware/Software Co-Design Spring /23 Simulation Based Verification vs. Formal Verification Both verification methods need a golden reference as part of the specification. Simulation based methods: stimuli (inputs) and monitors (expected outputs) Formal methods: mathematical model of desired properties Simulation-based methods are effective to catch bugs at early design stages, though only as good as the stimuli. Formal methods can provide a proof of correctness, but require more effort at design time. Simulation is still predominant while formal methods are catching up. More formal verification tools become available. More designers are trained to use these tools. e.g. most gate-level verifications are based on formal methods nowadays

11 ECE 587 Hardware/Software Co-Design Spring /23 Outline Verification Simulation Based Methods Equivalence Checking

12 ECE 587 Hardware/Software Co-Design Spring /23 Simulation Based Methods (Gajski et al.)

13 ECE 587 Hardware/Software Co-Design Spring /23 Test-Bench A test-case consists of a stimulus and the corresponding monitor. Monitors are usually generated from stimuli using a higher level model (golden) that is more likely to be correct. A collection of test-cases forms the test-bench. To achieve maximum productivity, each test-case should uncover some bug that has not been uncovered by a previous one waste of time to test part of DUT that have already been tested. How to measure the part being tested under a test-case?

14 ECE 587 Hardware/Software Co-Design Spring /23 Test-Bench A test-case consists of a stimulus and the corresponding monitor. Monitors are usually generated from stimuli using a higher level model (golden) that is more likely to be correct. A collection of test-cases forms the test-bench. To achieve maximum productivity, each test-case should uncover some bug that has not been uncovered by a previous one waste of time to test part of DUT that have already been tested. How to measure the part being tested under a test-case?

15 ECE 587 Hardware/Software Co-Design Spring /23 Test-Bench A test-case consists of a stimulus and the corresponding monitor. Monitors are usually generated from stimuli using a higher level model (golden) that is more likely to be correct. A collection of test-cases forms the test-bench. To achieve maximum productivity, each test-case should uncover some bug that has not been uncovered by a previous one waste of time to test part of DUT that have already been tested. How to measure the part being tested under a test-case?

16 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Could be based on the percentage of DUT that has been checked e.g. in terms of # lines of source code or # states in a state diagram modeling the system However, that s different from the entire behavior of the design. A single statement may involve many variables and a test-case covers it may miss some important corners. All corner cases may be modeled explicitly in the state diagram for coverage. But then the size of the diagram will become a big concern.

17 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Could be based on the percentage of DUT that has been checked e.g. in terms of # lines of source code or # states in a state diagram modeling the system However, that s different from the entire behavior of the design. A single statement may involve many variables and a test-case covers it may miss some important corners. All corner cases may be modeled explicitly in the state diagram for coverage. But then the size of the diagram will become a big concern.

18 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Could be based on the percentage of DUT that has been checked e.g. in terms of # lines of source code or # states in a state diagram modeling the system However, that s different from the entire behavior of the design. A single statement may involve many variables and a test-case covers it may miss some important corners. All corner cases may be modeled explicitly in the state diagram for coverage. But then the size of the diagram will become a big concern.

19 ECE 587 Hardware/Software Co-Design Spring /23 Performance Considerations Coverage may increase as you simulate with more test-cases. However, simulation takes time. Need to trade-off verification performance with quality Stimulus optimization: simulate less cases Use coverage feedback mechanism to improve test-cases that are otherwise generated randomly Not all test-cases are valid only simulate with valid ones Monitor optimization: discover bugs faster White box testing also monitor internal variables Use assertions instead of golden for internal variables Communicate more effectively with designers via visualization Speed-up techniques: faster simulation Faster algorithms Hardware assistance, e.g. on FPGA Simulate at higher abstraction levels, e.g. TLM, if certain details can be omitted.

20 ECE 587 Hardware/Software Co-Design Spring /23 Performance Considerations Coverage may increase as you simulate with more test-cases. However, simulation takes time. Need to trade-off verification performance with quality Stimulus optimization: simulate less cases Use coverage feedback mechanism to improve test-cases that are otherwise generated randomly Not all test-cases are valid only simulate with valid ones Monitor optimization: discover bugs faster White box testing also monitor internal variables Use assertions instead of golden for internal variables Communicate more effectively with designers via visualization Speed-up techniques: faster simulation Faster algorithms Hardware assistance, e.g. on FPGA Simulate at higher abstraction levels, e.g. TLM, if certain details can be omitted.

21 ECE 587 Hardware/Software Co-Design Spring /23 Performance Considerations Coverage may increase as you simulate with more test-cases. However, simulation takes time. Need to trade-off verification performance with quality Stimulus optimization: simulate less cases Use coverage feedback mechanism to improve test-cases that are otherwise generated randomly Not all test-cases are valid only simulate with valid ones Monitor optimization: discover bugs faster White box testing also monitor internal variables Use assertions instead of golden for internal variables Communicate more effectively with designers via visualization Speed-up techniques: faster simulation Faster algorithms Hardware assistance, e.g. on FPGA Simulate at higher abstraction levels, e.g. TLM, if certain details can be omitted.

22 ECE 587 Hardware/Software Co-Design Spring /23 Performance Considerations Coverage may increase as you simulate with more test-cases. However, simulation takes time. Need to trade-off verification performance with quality Stimulus optimization: simulate less cases Use coverage feedback mechanism to improve test-cases that are otherwise generated randomly Not all test-cases are valid only simulate with valid ones Monitor optimization: discover bugs faster White box testing also monitor internal variables Use assertions instead of golden for internal variables Communicate more effectively with designers via visualization Speed-up techniques: faster simulation Faster algorithms Hardware assistance, e.g. on FPGA Simulate at higher abstraction levels, e.g. TLM, if certain details can be omitted.

23 ECE 587 Hardware/Software Co-Design Spring /23 Performance Considerations Coverage may increase as you simulate with more test-cases. However, simulation takes time. Need to trade-off verification performance with quality Stimulus optimization: simulate less cases Use coverage feedback mechanism to improve test-cases that are otherwise generated randomly Not all test-cases are valid only simulate with valid ones Monitor optimization: discover bugs faster White box testing also monitor internal variables Use assertions instead of golden for internal variables Communicate more effectively with designers via visualization Speed-up techniques: faster simulation Faster algorithms Hardware assistance, e.g. on FPGA Simulate at higher abstraction levels, e.g. TLM, if certain details can be omitted.

24 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Feedback I (Gajski et al.) The logic gates could be a simplification of branch conditions. It could be impossible to generate a single test-case to cover all branches.

25 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Feedback I (Gajski et al.) The logic gates could be a simplification of branch conditions. It could be impossible to generate a single test-case to cover all branches.

26 ECE 587 Hardware/Software Co-Design Spring /23 Coverage Feedback II The additional test-cases can be found manually or automatically (by formal methods). (Gajski et al.)

27 ECE 587 Hardware/Software Co-Design Spring /23 Visualizing Bugs Look for most intuitive ways to visualize the bugs (Gajski et al.)

28 Hardware Emulation (Gajski et al.) Speed-up simulations that cannot be speeded-up otherwise. Usually when cycle-accurate behavior has to be simulated ECE 587 Hardware/Software Co-Design Spring /23

29 ECE 587 Hardware/Software Co-Design Spring /23 Outline Verification Simulation Based Methods Equivalence Checking

30 ECE 587 Hardware/Software Co-Design Spring /23 Equivalence Checking Make sure two designs are the same in term of functionality Ensure the correctness of implementations and optimizations Designers could make mistakes when implement a system manually. Automated software tools for synthesis and optimization may have bugs. What does it mean by the same functionality? Same output given same input?

31 ECE 587 Hardware/Software Co-Design Spring /23 Equivalence Checking Make sure two designs are the same in term of functionality Ensure the correctness of implementations and optimizations Designers could make mistakes when implement a system manually. Automated software tools for synthesis and optimization may have bugs. What does it mean by the same functionality? Same output given same input?

32 ECE 587 Hardware/Software Co-Design Spring /23 Equivalence Checking Make sure two designs are the same in term of functionality Ensure the correctness of implementations and optimizations Designers could make mistakes when implement a system manually. Automated software tools for synthesis and optimization may have bugs. What does it mean by the same functionality? Same output given same input?

33 ECE 587 Hardware/Software Co-Design Spring /23 Combinational Equivalence Checking What does it mean by the same functionality? Combinational circuits C1 and C2 At least they should have the same inputs and outputs For any input x, the output C1(x) should be the same as C2(x) One can build a larger combinational circuit E(x) = (C1(x) C2(x)). XOR each pair of corresponding bits of C1(x) and C2(x) Then OR the results together To prove C1 and C2 are equivalent is the same as to prove there is no x such that E(x) = 1. If such x exists, you can use it to debug your circuits.

34 ECE 587 Hardware/Software Co-Design Spring /23 Combinational Equivalence Checking What does it mean by the same functionality? Combinational circuits C1 and C2 At least they should have the same inputs and outputs For any input x, the output C1(x) should be the same as C2(x) One can build a larger combinational circuit E(x) = (C1(x) C2(x)). XOR each pair of corresponding bits of C1(x) and C2(x) Then OR the results together To prove C1 and C2 are equivalent is the same as to prove there is no x such that E(x) = 1. If such x exists, you can use it to debug your circuits.

35 ECE 587 Hardware/Software Co-Design Spring /23 Combinational Equivalence Checking What does it mean by the same functionality? Combinational circuits C1 and C2 At least they should have the same inputs and outputs For any input x, the output C1(x) should be the same as C2(x) One can build a larger combinational circuit E(x) = (C1(x) C2(x)). XOR each pair of corresponding bits of C1(x) and C2(x) Then OR the results together To prove C1 and C2 are equivalent is the same as to prove there is no x such that E(x) = 1. If such x exists, you can use it to debug your circuits.

36 ECE 587 Hardware/Software Co-Design Spring /23 Combinational Equivalence Checking What does it mean by the same functionality? Combinational circuits C1 and C2 At least they should have the same inputs and outputs For any input x, the output C1(x) should be the same as C2(x) One can build a larger combinational circuit E(x) = (C1(x) C2(x)). XOR each pair of corresponding bits of C1(x) and C2(x) Then OR the results together To prove C1 and C2 are equivalent is the same as to prove there is no x such that E(x) = 1. If such x exists, you can use it to debug your circuits.

37 ECE 587 Hardware/Software Co-Design Spring /23 Satisfiability (SAT) Find x such that E(x) = 1 for a given combinational circuit E or prove no such x exists. This is a well-studied problem: satisfiability (SAT) In theory, we don t know if there is a better way to solve it than to inspect the truth table of E. i.e. to try all possible inputs, 2 N for N inputs In practice, efficient solutions have been developed during the last two decades. That s the reason formal verification becomes more and more popular now.

38 ECE 587 Hardware/Software Co-Design Spring /23 Satisfiability (SAT) Find x such that E(x) = 1 for a given combinational circuit E or prove no such x exists. This is a well-studied problem: satisfiability (SAT) In theory, we don t know if there is a better way to solve it than to inspect the truth table of E. i.e. to try all possible inputs, 2 N for N inputs In practice, efficient solutions have been developed during the last two decades. That s the reason formal verification becomes more and more popular now.

39 ECE 587 Hardware/Software Co-Design Spring /23 Satisfiability (SAT) Find x such that E(x) = 1 for a given combinational circuit E or prove no such x exists. This is a well-studied problem: satisfiability (SAT) In theory, we don t know if there is a better way to solve it than to inspect the truth table of E. i.e. to try all possible inputs, 2 N for N inputs In practice, efficient solutions have been developed during the last two decades. That s the reason formal verification becomes more and more popular now.

40 ECE 587 Hardware/Software Co-Design Spring /23 FSM Equivalence Checking Concerning synchronous sequential circuits i.e. RTL designs For any input trace, the two FSM should produce the same output trace. From certain initial states What about states and state encodings? State encodings could be different. States could also be different. In general a very difficult problem that requires further research. In practice, if you could provide hints to relate the states of the two circuits, tools may be able to provide a proof.

41 ECE 587 Hardware/Software Co-Design Spring /23 FSM Equivalence Checking Concerning synchronous sequential circuits i.e. RTL designs For any input trace, the two FSM should produce the same output trace. From certain initial states What about states and state encodings? State encodings could be different. States could also be different. In general a very difficult problem that requires further research. In practice, if you could provide hints to relate the states of the two circuits, tools may be able to provide a proof.

42 ECE 587 Hardware/Software Co-Design Spring /23 FSM Equivalence Checking Concerning synchronous sequential circuits i.e. RTL designs For any input trace, the two FSM should produce the same output trace. From certain initial states What about states and state encodings? State encodings could be different. States could also be different. In general a very difficult problem that requires further research. In practice, if you could provide hints to relate the states of the two circuits, tools may be able to provide a proof.

43 ECE 587 Hardware/Software Co-Design Spring /23 FSM Equivalence Checking Concerning synchronous sequential circuits i.e. RTL designs For any input trace, the two FSM should produce the same output trace. From certain initial states What about states and state encodings? State encodings could be different. States could also be different. In general a very difficult problem that requires further research. In practice, if you could provide hints to relate the states of the two circuits, tools may be able to provide a proof.

44 ECE 587 Hardware/Software Co-Design Spring /23 FSM Equivalence Checking Examples Consider reachable states in product FSM (Gajski et al.)

45 ECE 587 Hardware/Software Co-Design Spring /23 A Special Case of FSM Equivalence Checking (Gajski et al.) With the same states, state encodings, and initial states, one just need to prove the next state and the output functions are equivalent. That s combinational equivalence checking. Solved!

46 ECE 587 Hardware/Software Co-Design Spring /23 A Special Case of FSM Equivalence Checking (Gajski et al.) With the same states, state encodings, and initial states, one just need to prove the next state and the output functions are equivalent. That s combinational equivalence checking. Solved!

47 ECE 587 Hardware/Software Co-Design Spring /23 Limitations of Combinational and FSM Equivalence Checking What if the functional equivalence goes beyond inputs/outputs per cycle? e.g. two kinds of processors with the same ISA but requiring different number of cycles to complete the same instructions? Although we perceive them as equivalent, it is difficult to define equivalence in mathematical sense. Equivalence checkings cannot be applied as of now. We have to apply a more expensive method called model checking.

48 ECE 587 Hardware/Software Co-Design Spring /23 Limitations of Combinational and FSM Equivalence Checking What if the functional equivalence goes beyond inputs/outputs per cycle? e.g. two kinds of processors with the same ISA but requiring different number of cycles to complete the same instructions? Although we perceive them as equivalent, it is difficult to define equivalence in mathematical sense. Equivalence checkings cannot be applied as of now. We have to apply a more expensive method called model checking.

49 ECE 587 Hardware/Software Co-Design Spring /23 Limitations of Combinational and FSM Equivalence Checking What if the functional equivalence goes beyond inputs/outputs per cycle? e.g. two kinds of processors with the same ISA but requiring different number of cycles to complete the same instructions? Although we perceive them as equivalent, it is difficult to define equivalence in mathematical sense. Equivalence checkings cannot be applied as of now. We have to apply a more expensive method called model checking.

50 ECE 587 Hardware/Software Co-Design Spring /23 Summary (Gajski et al.)