Automatic Debug If not now, when?

Transcription

1 Automatic Debug If not now, when? This presentation includes: Introduction to PinDown - an automatic debugger of regression tests News about PinDown 4, the next generation automatic debug, which was announced during the Design Automation Conference, June Product Overview PinDown - Automatic Debugger of Regression Tests 1

2 Verification-Debug is getting worse Year Engrs/Project % Time in Verification % % Verification Tasks Debug (37%) % More Complex Source: Wilson Research Group and Harry Foster at Mentor Graphics, 2014 Functional Verification Study Inputs e.g. random Ever Larger Designs and Design Teams Multiple Remote Design Teams Multiple IP Block Sources Incorrect Bug Reports SoC designs keep growing as we speak, so now we have more engineers contributing to the final SoC. When you find a bug in the system, it becomes a whodunnit show, and it takes a collaborative debug effort to go through piles of code and figure out what was going on. This is not just about the size it s about integrating multiple IP blocks representing multiple knowledge domains from multiple, remote teams. Shlomi Uziel VP of Engineering, Advanced Verification, CDN 2

3 The Principle is Simple......but the reality is difficult 41% Random instability occurs when a seed number does not produce the same test scenario because the DV testbench has had substantial changes. 41% incorrect bug reports with simple approach PinDown supports all scenarios The first commit that made a test fail is not necessarily the reason the test still fails. This only works in the simple case when one bug is inserted and fixed at a time, but in larger projects multiple bugs are introduced and fixed in parallel. 3

4 Automatic Debug with PinDown PinDown Customer PinDown s approach to automatically debug and send relevant bug reports directly to the engineers who made the mistakes is what you really need, rather than an endless list of failure reports to review. Using PinDown has allowed us to save valuable project time..senior Director of Hardware Design: Bugs Fixed 400% Faster 5X Less Thrashing (Discussion) Saved 4.9 Man-Years on 40 Engr Project 4

5 PinDown - Design Flow Interactions Bug Reports to Responsible Engineer Simulation Test Results PinDown Test Hub Version Control System PinDown Results Database Test Executor (LSF Farm) 5 10

6 How PinDown Works - Patented Debug Algorithm Smart Grouping 1-4X Debug Next Bug Bug Group 1: Error: RX value wrong 12 test failures Unroll Changes (10 selections in parallel) Test Fails Test Results * Tests Fastest Bug Group 2: Address is XXXXXXX 14 test failures Smart Selection Identify possible bad commits, files*, chunks*, lines* * New in PinDown 4 Test Passes Send Rerun Other Tests Check that unrolling the change makes all tests pass in the same bug group Tests Fastest Revision Control System (Perforce, GIT, Subversion, Clearcase) Bug No: 1 (new bug) Test: test34_vm_seed_0x Build: dtlb_thdr Error: Error: RX value wrong Committer: johndoe Commit Message: : RX performance improvement Committed Files: //depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv Log: /projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log PinDown makes the failing tests pass again, by only removing the changes made in one or several bad commits. This process, called validation, proves which commits are causing the respective test failures. As soon as a bad commit has been validated a bug report is sent to the committer. 6

7 Customer Benchmark Bug Fix Time: 400% Faster with PinDown s Regarding Bugs: 5X Less Discussion Manual Debug Manual Debug Automatic Debug Automatic Debug Hours Number of s Saved 4.9 Man-Years on 40 Engineer Project 7

8 Customer Quotes: Bug are fixed much faster with PinDown. PinDown pushes back issues to the engineers that caused the breakdown 24/7 so the quality is never allowed to slip..snr Principal Engineer You need to constantly supervise the quality to avoid slipping. If engineers don t get fast feedback and they start to discuss who should fix what then you lose a lot of time. Snr. Manager Using Pindown has allowed us to save valuable project time. It was a real game changer when we started to use PinDown..Snr. Director Automatic Debug...If not now, when? 8

9 Continuous Integration: - Gate keeper - Short directed tests - e.g. Jenkins Competitive Landscape Automatic Debug: Post-Integration Testing: - Large Test Suites - Random/Directed Tests PinDown Inhouse Solution

10 NEW! At DAC, June 2015, we announced information about the upcoming PinDown 4 release PinDown 4 The Next Generation Automatic Debug Finer Debug Granularity Debug granularity of files, chunks, lines Management Reports Track the cost of the computer farm related to the project activity in order to get the farm costs under control Track the bug fix time and quality uptime of the project Verification Optimizer Cut the farm cost, detect and find bugs faster 10

11 PinDown 4 : Fine Granularity Debug Smart Grouping Bug Group 1: Error: RX value wrong 12 test failures In PinDown 4 the debug granularity can be set to find which lines or files within a commit that caused a test to fail. This is useful on larger commits, e.g. mergers of two branches when a lot of changes have been done in one commit. 1-4X Unroll Changes (10 selections in parallel) Test Fails Debug Next Bug Test Results * Tests Fastest Bug Group 2: Address is XXXXXXX 14 test failures Smart Selection Identify possible bad commits, files*, chunks*, lines* * New in PinDown 4 Test Passes Send Rerun Other Tests Check that unrolling the change makes all tests pass in the same bug group Tests Fastest Revision Control System (Perforce, GIT, Subversion, Clearcase) Bug No: 1 (new bug) Test: test34_vm_seed_0x Build: dtlb_thdr Error: Error: RX value wrong Committer: johndoe Commit Message: : RX performance improvement Committed Files: //depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv Log: /projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log There is a trade-off between performance and granularity the finer granularity the longer the debug process takes. The granularity is controlled by the user, who can decide the optimal setting from project to project. 11

12 PinDown 4: Management Reports Debugger On Optimizer On Track farm cost per project, as well as bug fix time and quality uptime. You want to control and improve on these key performance indicators. Quality (test pass time ratio) Track Farm Cost Per Project Cost (tests run/bug) Bug Fix Time (from bug commit until verified fixed) Farm Waiting Times Time 12

13 PinDown 4: Verification Optimizer Finding Bugs Wasting Farm Resources and Debug Time Stop regression runs when all bugs have been found and save both time and computer resources. The stop time is calculated by analyzing historical runs to determine the optimal stop time. WITHOUT This cuts the farm costs, improves bug fix time as well as quality uptime. Bug 2 Bug 4 Bug 1 Bug 3 Debug Fix Bug 1 Comitted Bug 2 Comitted All Bugs Found Bug 3 Comitted Bug 4 Comitted WITH Reduced Farm Load Bug 2 Bug 1 Kill Jobs Bugs Fixed Faster Bug 3 Kill Jobs Bugs Found Faster Bug 3 Bug 4 Do a full run every X run to keep the coverage the same Debug Fix Debug Fix 13

14 Verifyter Roadmap PinDown 4.2 PinDown 4.1 Fine Granularity Lines Management Reports Across All Projects Verification Optimizer - Automated Fine Granularity Chunks Management Reports Per Project PinDown 4.0 Fine Granularity File Verification Optimizer - Tools Early Q Q Q