(Wasted) Effort Spent In Verification

I just spent a few minutes reading Harry Foster’s analysis of a functional verification study commissioned by Mentor Graphics and carried out by Wilson Research Group in 2010. There’s lots of good information in Harry’s analysis – there’s 9 posts in all – and I’d highly recommend people take a look if they haven’t already. It’ll be worth your time. The posts go back to June 26th, 2011… which, admittedly, makes me a little slow on the uptake :).

The absolute best piece of information in that series comes via my new favorite hardware development graphic. You’ll find it in part 5 of Harry’s analysis… or since I’ve pasted it below, you’ll also see it by scrolling down a touch.

Why do I like that graphic? Because it very clearly illustrates the near ridiculous amount of time we spend debugging code. In fact, if you compare the debug and testbench development slices of that pie, you see that we spend more time debugging code (33%) than we do writing it (28%).


If you’re interested in participating in a study that’s different-but-similar, I’d appreciated you completing this survey.


That should be an alarm bell for hardware developers – and I think it probably was for the people that saw it – but what do we do about it?

Seems we could handle this in a couple different ways. The first way would be to make more productive use of our time spent debugging. If we’re better at debugging, we spend less time debugging. Sounds great but I have a problem with that. The bugs have to come from somewhere. Just a guess, but they probably come from the people writing the code. Which means that what’s not immediately obvious in my new favorite graphic on (wasted) effort spent in verification is that a good portion of the 28% of the time verification engineers spend developing a testbench – and to be fair, the comparable amount of time designers spend creating a design – is dedicated to…

…creating bugs.

So how about this for a different direction… instead of finding faster ways to write then fix buggy code, how about writing more robust code in the first place? Better code… fewer bugs… less time spent debugging.

Serious.

If you want to know how to prevent bugs from getting into your code in the first place, you’ll want to go back to our Test-Driven Development posts from November last year. Those start here.

Say no to debugging. Write better code.

-neil

9 thoughts on “(Wasted) Effort Spent In Verification

  1. I knew when we introduced SystemVerilog as the “next big thing” that we were about to commit the same errors as the C++ and Java guys have, namely bloated languages that lead to enormous frameworks that are difficult to comprehend and no one can master.

    Where is the Python of the verification world? Why are we not writing testbenches in a concise, dynamic, language that promotes one way of writing verification frameworks with in-built syntax and very little template code – you know, when the manual says just copy these three lines as-is; it’s needed “behind the scenes” to allow fring to flobdicate correctly but you don’t have to know too much about flobdicating.

    Oh no! Someone has just advocated using a scripting language for writing testbenches. Before you laugh remember that every extra line of code, including template code, that you write has a debug and maintenance overhead. Dynamic languages are fast! You spend more time overall correcting and running bloatware than you would do if you wrote in a Python-esque TB language. A large part of the bloat in test frameworks seems to be a (poor) attempt at emulating the dynamism inherent in languages like Python. Many Scientists use dynamic programming environments like PythonXY and Enthought Python where a dynamic language links to C, C++ and Fortran libraries, allows compilation of a subset of Python for absolute speed or custom interfaces if necessary and maintain a high productivity – Think of what could be achieved if Python was the base of a TB generating environment!

    There is a gap in the market for and just as RubyOnRails and Python’s Django shook up the entrenched Web development market, I’d like to see similar happen for TB design.

    – Paddy.

  2. Hi Gordon, on PyHVL: Unfortunately it doesn’t present you with a verification methodology that is built-in to the language – you would have to write your own, admittedly in a better language than SystemVerilog though 🙂

  3. Not yet, but we are working on it. Already able to write verification environments in it, copying standard practices. Most of the existing verification methodologies are pretty much equivalent of plumbing. A useful part of the foundations of a house, but not really what you need to build the structure, anyway.

  4. Hi, I can’t agree more. Less code means less bugs and less debugging.
    There’s also one more thing. I prefer a “stricter” language which throws me a tonne of errors in the beginning, and tell me what those errors are. When I get those errors, they tell me exactly what’s wrong with my code, and they’ll be easy to fix. Compared with a less strict language that passes (and “propagates”) my errors to hardware / silicon, then I’ll have a hard time debugging later because the protocol / logic analyzers don’t (can’t) tell me where my errors are. One reason why I’m stuck with VHDL for so long… well, no offense, just my 2 cents.

    1. Hi Daniel.
      Python *is* strictly typed – meaning that it does throw an error if you try and do something foolish like add an integer to string.
      Python is not manifestly typed – meaning it is smart enough to work out that something is an integer without you having to add that information to every variable.
      Python is a dynamic language; it supports duck-typing. A lot of the dynamism necessary for verification has to be shoe-horned into statically typed languages and may break their type checking anyway. Some of the design pattern wizardry necessary for SystemVerilog are naturally supported in dynamic languages.

      P.S. some of the above comments do *not* apply to Perl – if Perl is the best scripting you have used so far, and Perl scripting seems to be the standard in verification apart from embedded TCL, then try Python – it is designed to be easy to both write and maintain code. The speed of prototyping some dynamic languages allow outside of the Verification field can lead to people junking the re-write in a “proper” static language phase and running the prototype. If only we could do the same (sigh)
      😉

Leave a Reply to Donald 'Paddy' McCarthy Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.