That’s what sets in after we’re done hitting all the easy stuff and move on to targeting the more obscure corners of a coverage model. The pace slows. Progress slows. We have lots of review meetings to debate the merits of coverpoints, some of which we may not even understand. Through trial-and-error we plod along as best we can until someone says whatever we have is good enough (because 100% coverage is impossible). Then we shrug our shoulders, add a few exclusions, write up a few waivers, shake off the hangover and move on.
I’ve had coverage hangover several times. I’m sure we all have. With some devices – the really massive SoCs – there are verification engineers that live through coverage hangover for months at a time. Their only reprieve, if you can call it that, tends to be bug fixing. If they’re lucky, they’ll get to implement a new feature now and then. Otherwise, it’s a cycle of regression, analysis, tweak and repeat.
The worst part of a coverage hangover is that the next hangover is guaranteed to be worse because the next device is always bigger. At least that’s what happens with current verification strategies. I’d like to propose we break common practice with a reset. Regrettably, it won’t change the fact that coverage space will continue to grow. But it will give some relief for the next few generations while folks smarter than me find better ways to define, collect and analyze coverage.
In closing coverage, our focus and timing have always struck me as being out of tune with the reality of the mission. I’m proposing we change that by breaking coverage into a series of steps that we can focus on independently, moving from one type to the next as features mature. Continue reading →
In publishing Portable Stimulus And Integrated Verification Flows, where I came up with the graphic that plots different verification techniques as a function of design scope vs. abstraction, I gave myself an opportunity to think more critically about verification than I have in the past. I’ve been recognizing different patterns and habits from my experiences and getting a better feel for how they’ve helped or hindered teams I’ve worked with. My most recent navel gazing has lead to a new and improved view for how design and testbench code matures over time; a way to quickly summarize quality in a way that’s meaningful and useful.
This is the same diagram I published a few weeks ago but with different classifications overlaid. I call them 5 steps in design maturity: broken, sane, functional, mature and usable. In my mind, every feature we create, every chunk of code we write, progresses through these 5 classifications on it’s way to delivery. Continue reading →
I’ve been writing a lot about integrated verification flows the last several weeks. I’ve had lots about how techniques fit together but I haven’t talked much about what holds everything together. This is where shared development objectives come in. Fundamental to an integrated verification flow is establishing shared objectives within the team. When I say shared objectives, I’m talking about objectives that represent tangible, demonstrable value at an integrated system level that require input across an entire team.
A test plan is a good place to establish shared objectives. In this post, we’ll work out a method for test planning that takes advantage of the dependencies between verification techniques, all for the purpose of creating tangible outcomes. Easy in theory but challenging in practice, it involves bottom-up application of a top-down test plan with the primary motivator being early and continuous delivery of product level outcomes.Continue reading →
Writing about portable stimulus has created some neat opportunities for me to swap ideas with others regarding how it fits into our verification paradigm. I like that. From a theoretical standpoint, I think I’ve heard enough to confirm my thought that it’s sweet spot is verifying feature sets with complex scenarios.
Trickier is to move beyond the theory toward recommendations for how it all fits together. By ‘all’ I mean more than just portable stimulus. I mean how all our verification techniques fit together in a complementary way as part of a start to finish verification flow. This is a first attempt at qualifying what ‘all’ looks like. I’ll go through each verification technique, comment on it’s purpose, how teams transition to/from it and discuss how it feeds subsequent verification activity.Continue reading →
All verification techniques can be effective given the right scope and applied abstraction. At least that’s the argument I started in Portable Stimulus and Integrated Verification Flows with a graphic that plots the effectiveness of several techniques as a function of scope and abstraction. I have more people agreeing with the idea than disagreeing so I’ve carried on with it. I decided to dig a little deeper into exactly where and how well techniques apply.Continue reading →
While a lot of information is produced to introduce and support individual verification techniques, methods for applying a variety of verification techniques in a complementary way are harder to come by. What’s here is a push in that direction; a rough guideline covering multiple techniques with suggestions for where each is most effective and how they integrate to verify a complete SoC.
The idea of building a verification flow isn’t at all new, but the possibilities for and constraints of such flows naturally change as new ideas emerge. The event that instigated this particular article was the release of the Early Adopter Portable Stimulus Standard. Long story short: with all the excitement around portable stimulus, I haven’t seen anyone paint a clear picture of how portable stimulus (a) integrates with or (b) replaces existing verification flows. To fill that void, I’ve mapped a set of possibilities for a generalized verification flow – one that includes portable stimulus – as a reference point.
I used to drive a piece of garbage green 2000 Civic sedan. It sipped gas, was super reliable on the days it would start, and it didn’t matter how I abused it because it was already as bad as a driveable car could get. It was perfect for a guy like me who couldn’t care less about the vehicle he drove.
In November last year we bought a trailer. We needed a vehicle to pull it so I cashed in the crap-mobile (that’s what my kids called it) for a $50 tax receipt and upgraded to a 2004 4Runner. Somewhat unexpectedly, I’ve turned into a “truck guy” in the 9 months since and my dream vehicle has become a new 4Runner TRD PRO.
A 2017 Toyota 4Runner costs $44,440; that’s the TRD PRO model with the roof rack and alloy wheel options. Because my aversion to spending money is stronger than my love for dream vehicles, I’ve decided I can’t afford that. But I’d still like to drive one so I have to find a way other than paying for one myself. I think I found it… Continue reading →
I generally consider myself too practical – aka: skeptical – to think that one small change in mindset is enough to motivate major change in performance. But this is one that I think deserves attention because the potential outcome could be significant.
How much time passes between writing a line of RTL or testbench code and knowing that it works. I know it varies, but on average how long would it take? A line of code is written, time passes, then a test is run that either verifies it’s correct or discovers a bug. Does that time pass as minutes? As days? Maybe weeks? Months?
I’m preparing for a talk at DAC next week in the Verification Academy booth and in going through my material I’ve started asking myself that question. It’s the 3rd year in a row the folks at Mentor have invited me for a session in the booth. The last 2 years have been a lot of fun and I’m hoping for the same this year. This year’s session is called Add Unit Testing To Your Verification Toolbelt. As the name suggests, the focus will be on how we can use unit testing and test-driven development to improve the quality of the hardware we build.
I believe quite strongly in the value of unit testing and test-driven development because the quality of what I produce as a verification engineer is noticeably better because of them. I believe so strongly in TDD in particular that I know I get a little preachy at times. It’s the most important engineering skill I’ve ever learned and I don’t shy away from telling people about it.
I always think of rigour first when it comes to benefits. Testing code as you write it a few lines at a time is a rigorous process to say the least. But during the run-up to DAC I’m starting to think it’s timing that’s the real key. With TDD, the amount of time that passes between writing and testing a line of code is minutes, sometimes less. A design and test cycle measured in minutes is almost too short a time to think about anything else. I focus on getting exactly 1 thing right. There’s no thinking back, no back-and-forth with a teammate that found a bug, flipping through my logbook for clues as to what I’ve done or editing bug reports. In fact, with TDD there’s hardly any context switching at all. I write a line or 2, test them, if they’re broken I fix them, then I move to the next line. Most of the bugs I create are killed immediately; which coincidentally is the best time to kill them.
I’ve seen a lot of people talk about how bugs become more costly to fix the longer they live. For me, TDD has been the most effective way to respond.
If you’re at DAC next week in Austin and using TDD to kill bugs fast sounds good – or great! – I hope to see you at Mentor’s Verification Academy Booth at 11am on Monday. It’ll be a half hour presentation with lots of time for discussion. We’ll talk about TDD, unit testing, SVUnit and all the frustration that magically vanishes because of them!