Ad
More Related Content
What's hot (20)
Similar to Verification Bug Metrics: A Different Approach (20)
Ad
More from DVClub (20)
Ad
Recently uploaded (20)
Verification Bug Metrics: A Different Approach
- 1. <Insert Picture Here> Verification Bug Metrics A Different Approach Greg Smith [email protected]
- 2. 2 Talk Outline • The hardest question for a DV manager to answer • How does chip design use metrics today? • How does Software do it? • Adapting SW metrics principles to HW design • Actual project data
- 3. 3 Some Background • Became a design verification manager at HP • With technique I am presenting today, successfully taped out 10 ASICs with first pass success – Some data from those projects is presented here • The hardest question for a DV manager to answer:
- 4. 4 What do you think it is? • The hardest question for a DV manager to answer: When will you be done?
- 5. 5 Metrics Used Today • What are the most common metrics in use today to track execution and measure progress: > Completion of Test Plan > Functional & Code Coverage percentages > Bug Arrival Rates > RTL rate of change > Schedule milestones • The above are all backwards looking and subject to human error of omission! • I was searching for a quantitative, *calculated* way to predict and track schedule.
- 6. 6 Cumulative Bugs Time (weeks) NumberofBugs Cumulative Bugs Does this bug arrival chart tell you that you are ready for Tape out? HP Project E
- 7. 7 Cumulative Bugs closed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Time (Weeks) BugCount Cumulative Bugs closed Does this bug arrival chart tell you if you are on schedule? What does it tell you about your staffing level? HP Project M
- 8. 8 How Does Software Do It? • Researched web for Software Development Metrics • Software +defect +reliability +metrics • Amazing amount of material: PMI, SEI, University and government research • Reference Holly Richardson :“Rayleigh Curve Based Estimation” • Discovered many corroborating papers and studies on related topics • How many resources are needed, when, and for how long • Bug arrival curve predictions • Technique I am presenting today was used on the software project for the NASA space shuttle
- 9. 9 The Rayleigh Distribution Model So what is all this? Em = errors expected in this period td = Total # of measurement periods t = elapsed time (This measurement period). Er = total # of errors expected • Therefore you need 2 pieces of information: Project Duration # of bugs expected 2 2 3 2 m te 6 =E dt t d r t E − ×
- 10. 10 Applying the Principle to HW Design Project Duration (td) • # of measurement periods between when you start counting bugs and when you freeze or tape out or equivalent milestone • Typical would be weeks t term is the given week 2 2 3 2 m te 6 =E dt t d r t E − ×
- 11. 11 Predicting the # of Bugs to be Found: What is your bug density? (Er) • How I went about figuring this out: – SW Uses bugs per LOC or KLOC – Went through bug data base for several past projects – Total up all RTL bugs – Sized the corresponding design (lines of Verilog code) – Observed a trend which established my initial “guess” – Our density was 1/150 LOC for all projects • SW norms are 1/50 – 1/100 2 2 3 2 m te 6 =E dt t d r t E − ×
- 13. 13 Something WAS Wrong Predicted Bug Arrival Cumulative Logic Bugs Test Bench Fixed
- 14. 14 Cumulative Bugs closed 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Cumulative Bugs closed Does this bug arrival chart tell you if you are on schedule? What does it tell you about your staffing level?
- 15. 15 Something is Wrong Here!
- 16. 16 0.0 5.0 10.0 15.0 20.0 25.0 Weekly Arrivals - Predicted Using Projections for Planning
- 17. 17 0.0 5.0 10.0 15.0 20.0 25.0 Weekly Arrivals - Predicted • Can tell you how many resources you need • To find max # bugs/week = how many resources? • Can help you plan release points • Go to emulation when 75% of bugs are found • Start freeze process when 90% of bugs are found Using Projections for Planning
- 18. 18 Chip Verif Progress "at a glance" 0 5 10 15 20 25 30 35 BugCount 0 10 20 30 40 50 60 70 80 90 100 Coverage% Cumulative Logic Bugs Found Predicted Bug Arrival Line Coverage % Condition Coverage %
- 19. 19 Is this a one trick pony? • HP Results are for ASICs & large FPGA • Taped out 10 bug free chips that followed the model to a high degree • Robert Page applying this technique at Freescale • Kim Asai (TI) and Rahman Farhan (AMD) expressed interest • What about processor projects? • Researched past projects at Oracle to see: > Does the arrival rate equation match? > What can I use to calculate bug density? • Applied to a recently completed processor project with success
- 20. 20 SOC Blocks Arrivals vs Predicted
- 21. 21 Ultra SPARC T2 Core Actual vs Predicted Arrivals
- 22. 22 Bug Density in Processor Projects • Cant use bugs / Line. Structural coding style makes using lines of code problematic • What else measures design size? • Code coverage metrics gave me an idea: # of Conditions, as reported by condition coverage metric might be usable • Discovered decent correlation amongst projects
- 23. 23 Actual chart from recent processor 0 Actual Bug Arrivals vs Predicted BugCount • Predicted bug count was off by less than 1%
- 24. 24 How can I use this: Determining your bug density factor. Bugs per ??? • If you have Behavioral Verilog – use LOC • Applying to a different design/coding style will require work • Research past projects and look at historical bug density • Bottom line – you need to do some homework but you can find something that you can work with
- 25. 25 Conclusion • How to answer that difficult question? • In the SW world, I found a technique I was looking for: > Quantitative– calculated value > Predictive – am I on track? When might I be done? • Rayleigh arrival model borrowed from software • To use this technique you need to: > Assess your own RTL team’s defect density - Use past project’s historical data
- 26. 26 Conclusion • You can create a viable method that can be used to track your bug discovery rate vs. an expected rate • This provides a point of reference that you can use to assess the state of your project. • Combined with traditional metrics, the technique can contribute to your assessment of “Am I Done?”, but also help answer the question: When will you be done?
- 27. DV Club Presentation Slide 27 Thank you for your attention.