39% of Your Verification Budget Goes to What???

I used to drive a piece of garbage green 2000 Civic sedan. It sipped gas, was super reliable on the days it would start, and it didn’t matter how I abused it because it was already as bad as a driveable car could get. It was perfect for a guy like me who couldn’t care less about the vehicle he drove.

In November last year we bought a trailer. We needed a vehicle to pull it so I cashed in the crap-mobile (that’s what my kids called it) for a $50 tax receipt and upgraded to a 2004 4Runner. Somewhat unexpectedly, I’ve turned into a “truck guy” in the 9 months since and my dream vehicle has become a new 4Runner TRD PRO.

A 2017 Toyota 4Runner costs $44,440; that’s the TRD PRO model with the roof rack and alloy wheel options. Because my aversion to spending money is stronger than my love for dream vehicles, I’ve decided I can’t afford that. But I’d still like to drive one so I have to find a way other than paying for one myself. I think I found it…

Harry Foster finished up posting his results and analysis from his latest Functional Verification Survey in January. He’s done that every couple years going back to 2010. The 2016 version was great as usual. It had my favorite semiconductor development statistic:

In isolation this is bad; the fact verification engineers now spend 39% of their time debugging code. Thanks to Harry we have enough data to also look at the debug trend which is worse than bad… but also encouraging for someone who can’t affort a new 4Runner.

In 2010, SoC teams spent the equivalent of 0.77 4Runners on debug for each verification engineer. Debug spending has been increasing steadily over a six year period to the point we’ve reached 0.94 4Runners/verification engineer.

That’s a large expense but for one verification engineer this isn’t too bad. I can’t drive 0.94 of a 4Runner and I’m not going to change my coding habits for something I can’t drive.

But that’s me being selfish. Let’s look at this from a team spending perspective to see the big picture impact.

Yikes! These are major expenses. A verification team of 5 is spending 4.71 4Runners every year debugging code. If you’re leading a team of 40 verification engineers you’re paying 37.67 4Runners. The sucky part, in case it’s not obvious, is that you’re only paying for 4Runners; you don’t actually get them. Not a great value. Paying for all those 4Runners is a major expense, and you’re paying for them every year, in increasing numbers. In fact, paying for 4Runners you don’t get is your verification team’s largest expense.

If we extrapolate to 2018, it’s quite likely we’ll surpass 1.00 4Runners/verification engineer wasted on debug. This is terrible news for the people signing the cheques but great news for verification engineers that want a 4Runner they can’t afford. At 1.00 4Runners/verification engineer we reach the point companies are fine with wasting the equivalent of a new 4Runner for every verification engineer every year. That means the 40 person verification team will budget for 40 4Runners it never gets.

Depressing… but there is a silver lining to budgeting for 40 4Runners you never get.

If a team of 40 reduces the amount of effort it spends on debug by a measly 2.5%, it actually gets 1 whole 4Runner. A 5% reduction gets you two 4Runners. Real ones! And so on. There’s your silver lining. The amount of money we spend on debug is so huge that a small single digit reduction is enough to buy some lucky team member their dream vehicle. Ideally, I’m on this team and, fingers crossed, I get my new 4Runner.

I’ll sum up this moderately ridiculous post with two serious questions to verification leadership. First: knowing that even small reductions in debug effort could mean massive savings and that everything we’re doing now has us trending in the wrong debug direction, what will it take to put a new 4Runner TRD PRO with a roof rack and alloy wheels in your parking lot for real? Second: if you think think you can do it, can I be on your team 🙂 .


PS: I used an average salary of $107,300 for verification engineers. That’s a stat I found a few years ago so it’s probably a bit low in today’s dollars. More than likely, our 4Runners/verification engineer has already reached the all important 1.00 to get me new wheels.

2 thoughts on “39% of Your Verification Budget Goes to What???

  1. The salary for an engineer is the “cheap part”. There’s any number of overhead costs (lights, rent, computer, fractions of managers, HR people, facilities people). Then there’s the cost of the tools and maintenance (which can be really huge). In the early 90s I was told to use the number of $150/hour as the cost per engineer.

    But the REALLY huge cost is the opportunity cost: how many sales did the company lose by being several months later on verification? If you’re first to market, often you wind up being the de-facto standard, even if your design is buggy (Broadcom’s strategy for decades). Or put it another way: if the company could just “write a check” and have verification instantly done, how much would it be worth to them? that’ll tell you the true cost of each hour of verification.

    Since unit-testing and code-reviews are both the most effective and cheapest way to reduce debug and simulation, those are the first things to do.

Leave a Reply

Your email address will not be published. Required fields are marked *