How many people in the world are named Glenn? An odd question for sure, but what if that were your job, to verify the number of people named Glenn in the entire world? How would you do it?

Option 1:

Your first choice could be to walk up to 20 different strangers and ask them if their name is Glenn. If 1 of them says yes, you could assume that 5% of the people on earth – 1 in 20 – go by the name Glenn. Multiply the number of people on earth by 5% and there’s your answer. Should anyone have the nerve to challenge your conclusion, you could simply ask those same 20 people again – maybe even ask them several times just to be sure.  Assuming no one has changed their name to disrupt the results, you could indeed confirm that 5% of the people in the world are named Glenn. Done.

Option 2:

How about instead you walk up to absolutely everyone on the planet and ask them if their name is Glenn. If that’s what you need to know, then that’s the fool proof way to do it…

…though obviously that’d take a while, so instead maybe you could phone 1 person in every country, delegate the hard part (counting the Glenns), have them call you back in a few days and then summarize the results. Yes, a reporting and data collection hierarchy would make it easier so you could go with that…

…but what about language? You might only speak english so calling every country in the world may be a problem, never mind the long distance charges. How about an email? You can send an email with a simple instruction that says “ask all the people in your country if their name is Glenn”. All your data collection people could respond by email, you tally the results and presto, you know exactly how many Glenns live on earth.

Uh-oh. What if some people don’t have email? Then what? If they don’t have email, they may prefer to go old-school with a fax. Or maybe phone is still best in a few cases, though you’d probably need an interpreter… or I guess several interpreters just in case. Or maybe you can send an email to a friend of theirs who could pass instructions and results between the 2 of you, like a bridge. You may also need to have a primary and backup means of communication just to be safe. You do need to count all the Glenns, remember.

And hey! If you’re counting the number of Glenns in the world, why not count the number of Ricks as well? You may not need to know how many Ricks there are now, but someday that might be useful. Whoa! Here’s an idea… you could generalize the instructions into “ask all the people in your country if their name is _______”. With a separate correspondence – a sideband data record of sorts – to let people know what name you’re interested in, you’ve got a recipe for counting any name you want!

No! Not good enough! You can do better! What if sometime down the road some one wants to count something else? What if they want to count roosters named Glenn? Or cows named Betsey? Or hippos name Snuggle Face? No problem! If you generalize your instructions into “ask all the ______ in your country if their name is _______”. Then you could send a follow up that details what you’re counting and another to tell people the name. Only 3 simple transactions to count anything named anything! Beautiful and reusable!

Still not good enough. Reusable is good but reusable and extensible is better. This would be far more efficient if you could pick some better way to split things up than countries. Some countries are enormous (Glenn’s in China anyone?) so it might be nice to have someone in every city instead of 1 person for the whole country. Or maybe 1 person for the country that could talk to a person in each province or state who could then talk to people in cities and towns (i.e. “ask all the ______ in your ______ if their name is ______”). They could all do a pre-summary, pass them up the chain and you could summarize the summaries (though of course you’d want to raw data as well, just in case). Yes, more hierarchy and filtering of results would be far more efficient and extensible. Of course you’d need a format for the pre-summary and aggregated pre-summaries, and it’t be good to have a few options for those with scripting to convert from 1 format to another.

Ok. You’ve got it. Different instructions that could be used to count the names of people, cows or anything else you might be interested in with a plan and hierarchy for disseminating those instructions to both highly and sparsely populated countries, states/provinces, cities and towns over several different communication mediums with back-up communications in any language and several defined formats for the pre-summary of results and a way to merge data from all these formats. Done…

…unless of course you want to be flexible enough to count the number of Glenns in the world whose mother’s neighbour’s grandmother’s dog’s name if Fido. You can’t do that now… but you could add it!

What’s The Better Approach?

That depends on whether you want bogus results in 10 seconds or reams of incomprehensible data in 2 years.

Neither, obviously, are any good. So why the heck then would I write all this gibberish? Because I think these 2 options represent tactics that verification engineers regularly choose between to verify hardware (yes, we do. I’ve used both… more than once :().

Option 1 is like running an entire suite of directed tests with the same packets, instruction set, data alignment, random seed, link utilization, transaction timing, etc over and over again thinking we’re doing something useful. Or like having 1 directed test for flow control that never varies the interaction between master and slave. I’m sure most verification engineers have written tests like these before. They run on the big server farms organizations are growing to get through regressions in a week or less.

I’m just as sure that most of us have gone to the other extreme – the reusability/extensibility rat hole in option 2 – where we think we need to see absolutely every possible state in a design from every possible direction before we can be confident all is well. And to do that, we build ridiculously complex constrained random environments that are so reusable and so extensible that they’re not actually usable at all (at least not by people other than the person that built it).

I think falling into the trap of aligning with either of these approaches comes from us overlooking the fact that that their is no repeatable recipe for functional verification; no one approach or set of techniques is going to work every time; no amount of planning and navel gazing is going to be enough to accurately predict the future. And sure what I have here may seem a little goofy, but close your eyes and think back to past projects you’ve been a part of. I’d be willing to bet the 2 options I have here are more common than we’d like to believe.

They shouldn’t be… but I’m sure they are.

-neil

P.S. I forgot to mention that standardizing either of these approaches wouldn’t make them any better.