Better Luck Next Time

I know. It’s been a while. I’ve been dealing with something, something big, and it’s taken me a while to come to terms. But after several months, I’m finally in a place where I can talk about it.

I have something incredibly disappointing to report. It crushes me to be the one to put it out there but it’s the right thing to do. We can’t hide from it or pretend it didn’t happen. Facing it head on is the only way. It’s no one’s fault. None of us could have foreseen it. Personally, I was blindsided by it. We were destined for a glorious milestone we could rally around; celebrate even. A new tomorrow of off-road adventures; days spent blazing through untamed wilderness on TRD tuned suspension; sunshine on the horizon, dust in our wake.

Alas. It wasn’t meant to be.

A couple years ago, the 2016 Wilson Research Group Verification Study – commissioned by Mentor and, of course, lead by Harry Foster – confirmed a clear trend that goes back to at least 2010: the amount of time verification engineers spend doing debug has been steadily increasing for the better part of a decade.

This trend is troubling. To me it’s become an indication of a fundamental flaw in how we approach verification. Despite our advancements over the same period, bugs are chewing up a growing portion of our time. Not good.

But as troubling as it is, in 2016 I recognized the trend wasn’t all bad because it presented verification engineers with an incredible opportunity.

Through a simple conversion of percentages into cost of debug as salaries paid, I saw that we were steadily approaching an amount equal to the cost of a brand new vehicle; specifically, the cost of a Toyota 4Runner TRD PRO (with the roof rack and 17-in alloy wheel packages). In 2010, teams were budgeting 77% of the cost of a new Toyota 4Runner on debug. By 2016 – the time of my epiphany – we were at 94%.

Coincidentally, when you extrapolate a data point for 2018 using the average increase over the previous 6 years you get 1.00. That’s 100% of the purchase price of a Toyota 4Runner (with the roof rack and 17-in alloy wheel packages) budgeted every year for every verification engineer.

100% is a big number.

To the opportunity… with so much money spent on debug I think it’s in a company’s best interest to incentivise verification engineers to reduce the expense. Further, I think a special award should go to verification engineers able to reduce their debug time to 0. That special award could be, for example, a Toyota 4Runner TRD PRO (with the roof rack and 17-in alloy wheel packages).

For those of you thinking this sounds like crazy talk, think again about what you’re getting: a dramatic improvement in product quality with a likely reduction in development time from engineers that are much happier, all at no extra cost.

Not so crazy anymore… except here comes the disappointment.

Before you book a seat on the bandwagon I’ll remind you that I predicted this based on the 2016 data. Fast-forward to the 2018 and sadly it all falls apart.

Turns out my prediction for 2018 was a little too optimistic. Published earlier this year, the actual 2018 data show that while we’ve done our part – verification teams now budget 44% of their time on debug – Toyota also increased their MSRP. Combined, those increases take us to only 97.2% of a Toyota 4Runner. In other words, we’re $1,363 worth of debug short. Even if we remove the alloy wheel package (the roof rack comes standard on the 2019 TRD PRO), we’re still $113 short. In summary: no 4Runner.

I know. It hurts.

Let’s all hope the 2020 verification survey brings better news.

-neil