I’m anti-constrained random. I would never use constrained random. I think it’s an overhyped technique that doesn’t produce the results we think it does. People would be better off forgetting about it and going back to directed testing.

Ok… a little strong perhaps, but if you’ve read my posts (aka: rants) on constrained random verification, you may be assuming that’s what I think of it; that constrained random is something I’m opposed to and that I’ve turned back the clock to the directed testing stone ages.

Not true.
Continue reading →

If you’ve read Does Constrained Random Verification Really Work and Functional Verification Doesn’t Have to be a Sideshow, you’ll know that I’ve become a bit of a skeptic when it comes to constrained random. My opinion hasn’t changed much since those posts and I think I’ve got a couple visuals that will help people see the point I was arguing in Functional Verification Doesn’t Have to be a Sideshow, that a successful constrained random verification effort starts with directed testing… a lot of directed testing. Continue reading →

This is another one of those challenge-the-way-we-think-about-functional-verification posts. The motivation behind it comes from a few different places.

First is a blog posted on eetimes designline by Brian Bailey a couple weeks back called Enough of the sideshows – it’s time for some real advancement in functional verification! In that post, Brian exposes a few techniques – constrained random verification in particular – that have failed to live up to the hype and praise that’s been heaped upon them over the last several years. That’s an industry expert suggesting we re-think the direction we’re going with functional verification tools so if you haven’t read it, I’d suggest doing so. In case you missed it, Brian also inspired the snappy title of this post :). Continue reading →

Being that we’re a week away from TDD month on AgileSoC.com, I thought that an appropriate way to get people thinking in a different direction – yes… test-driven development would take us in a different direction – would be to dig up a functional verification article from a couple years ago that I co-wrote. A good part of the article focused on legacy process and it opens by taking a few shots at constrained random verification.

Constrained random verification is pretty mainstream in ASIC and FPGA verification these days, though it does mean different things to different teams. The argument for constrained random verification has always been that it’s a more productive way to cover the state space in increasingly large and complex designs. I used to believe that wholeheartedly. But after seeing and hearing about it fall short – from an efficiency point of view – many times, my current impression of constrained random verification is that it just doesn’t work as well as we all want it to.

Continue reading →

Exploratory testing is a term I heard several times at Agile2011 in Salt Lake City a couple weeks ago. As I heard people talking about it, exploratory testing seemed to be something that verification engineers should be familiar with. Admittedly, it was new to me so for anyone else new to exploratory testing, here’s a description from good old wikipedia:

Exploratory testing seeks to find out how the software actually works, and to ask questions about how it will handle difficult and easy cases. The quality of the testing is dependent on the tester’s skill of inventing test cases and finding defects. The more the tester knows about the product and different test methods, the better the testing will be.

From that definition and other things I’ve read since Agile2011, it seems that exploratory testing is more easily pondered relative to it’s antithesis: scripted testing. In scripting testing all the thought is put in upfront; a tester would do all their research first, build the test plan and then they or someone else would execute the test plan. With exploratory testing, the tester assumes its impossible to deduce everything up front and that the goals of the testing will change over time. He/she starts with a more basic plan and then thinks their way through the test process, learning and shaping the test plan as they go along.

Continue reading →