Balancing Verification Development With Delivery

Verification engineers have a habit of over-engineering testbenches and infrastructure. We can all admit that, no? I can certainly admit it. From first hand experience, I can also admit that the testbenches I’ve over-engineered had a lot of waste build into them; unnecessary features I forced people to use and features that were never used at all. And there’s no good reason for it other than I like building new testbench features. Don’t think I’m alone there.

We’re overdue for a shift in verification. We need to de-emphasize the value of testbench and infrastructure development and re-emphasize the value of delivery. Here’s how we get started. Continue reading

A Cure For Our Coverage Hangover

Coverage hangover.

That’s what sets in after we’re done hitting all the easy stuff and move on to targeting the more obscure corners of a coverage model. The pace slows. Progress slows. We have lots of review meetings to debate the merits of coverpoints, some of which we may not even understand. Through trial-and-error we plod along as best we can until someone says whatever we have is good enough (because 100% coverage is impossible). Then we shrug our shoulders, add a few exclusions, write up a few waivers, shake off the hangover and move on.

I’ve had coverage hangover several times. I’m sure we all have. With some devices – the really massive SoCs – there are verification engineers that live through coverage hangover for months at a time. Their only reprieve, if you can call it that, tends to be bug fixing. If they’re lucky, they’ll get to implement a new feature now and then. Otherwise, it’s a cycle of regression, analysis, tweak and repeat.

The worst part of a coverage hangover is that the next hangover is guaranteed to be worse because the next device is always bigger. At least that’s what happens with current verification strategies. I’d like to propose we break common practice with a reset. Regrettably, it won’t change the fact that coverage space will continue to grow. But it will give some relief for the next few generations while folks smarter than me find better ways to define, collect and analyze coverage.

In closing coverage, our focus and timing have always struck me as being out of tune with the reality of the mission. I’m proposing we change that by breaking coverage into a series of steps that we can focus on independently, moving from one type to the next as features mature. Continue reading

5 Steps In Design Maturity

In publishing Portable Stimulus And Integrated Verification Flows, where I came up with the graphic that plots different verification techniques as a function of design scope vs. abstraction, I gave myself an opportunity to think more critically about verification than I have in the past. I’ve been recognizing different patterns and habits from my experiences and getting a better feel for how they’ve helped or hindered teams I’ve worked with. My most recent navel gazing has lead to a new and improved view for how design and testbench code matures over time; a way to quickly summarize quality in a way that’s meaningful and useful.

This is the same diagram I published a few weeks ago but with different classifications overlaid. I call them 5 steps in design maturity: broken, sane, functional, mature and usable. In my mind, every feature we create, every chunk of code we write, progresses through these 5 classifications on it’s way to delivery. Continue reading

Verification Planning From The Top Down

I’ve been writing a lot about integrated verification flows the last several weeks. I’ve had lots about how techniques fit together but I haven’t talked much about what holds everything together. This is where shared development objectives come in. Fundamental to an integrated verification flow is establishing shared objectives within the team. When I say shared objectives, I’m talking about objectives that represent tangible, demonstrable value at an integrated system level that require input across an entire team.

A test plan is a good place to establish shared objectives. In this post, we’ll work out a method for test planning that takes advantage of the dependencies between verification techniques, all for the purpose of creating tangible outcomes. Easy in theory but challenging in practice, it involves bottom-up application of a top-down test plan with the primary motivator being early and continuous delivery of product level outcomes. Continue reading

Verification That Flows

Writing about portable stimulus has created some neat opportunities for me to swap ideas with others regarding how it fits into our verification paradigm. I like that. From a theoretical standpoint, I think I’ve heard enough to confirm my thought that it’s sweet spot is verifying feature sets with complex scenarios.

Trickier is to move beyond the theory toward recommendations for how it all fits together. By ‘all’ I mean more than just portable stimulus. I mean how all our verification techniques fit together in a complementary way as part of a start to finish verification flow. This is a first attempt at qualifying what ‘all’ looks like. I’ll go through each verification technique, comment on it’s purpose, how teams transition to/from it and discuss how it feeds subsequent verification activity. Continue reading

A Verification Where And How

All verification techniques can be effective given the right scope and applied abstraction. At least that’s the argument I started in Portable Stimulus and Integrated Verification Flows with a graphic that plots the effectiveness of several techniques as a function of scope and abstraction. I have more people agreeing with the idea than disagreeing so I’ve carried on with it. I decided to dig a little deeper into exactly where and how well techniques apply. Continue reading

Portable Stimulus And Integrated Verification Flows

While a lot of information is produced to introduce and support individual verification techniques, methods for applying a variety of verification techniques in a complementary way are harder to come by. What’s here is a push in that direction; a rough guideline covering multiple techniques with suggestions for where each is most effective and how they integrate to verify a complete SoC.

The idea of building a verification flow isn’t at all new, but the possibilities for and constraints of such flows naturally change as new ideas emerge. The event that instigated this particular article was the release of the Early Adopter Portable Stimulus Standard. Long story short: with all the excitement around portable stimulus, I haven’t seen anyone paint a clear picture of how portable stimulus (a) integrates with or (b) replaces existing verification flows. To fill that void, I’ve mapped a set of possibilities for a generalized verification flow – one that includes portable stimulus – as a reference point.

Continue reading

39% of Your Verification Budget Goes to What???

I used to drive a piece of garbage green 2000 Civic sedan. It sipped gas, was super reliable on the days it would start, and it didn’t matter how I abused it because it was already as bad as a driveable car could get. It was perfect for a guy like me who couldn’t care less about the vehicle he drove.

In November last year we bought a trailer. We needed a vehicle to pull it so I cashed in the crap-mobile (that’s what my kids called it) for a $50 tax receipt and upgraded to a 2004 4Runner. Somewhat unexpectedly, I’ve turned into a “truck guy” in the 9 months since and my dream vehicle has become a new 4Runner TRD PRO.

A 2017 Toyota 4Runner costs $44,440; that’s the TRD PRO model with the roof rack and alloy wheel options. Because my aversion to spending money is stronger than my love for dream vehicles, I’ve decided I can’t afford that. But I’d still like to drive one so I have to find a way other than paying for one myself. I think I found it… Continue reading