DVCon call for papers hit my inbox last week. This time around, to possibly save myself the effort, I figured it might be more productive to let other people decide whether or not mine is a topic they’d like to see at DVCon. I’ve got a draft title and abstract so far…
How UVM-1.1d Makes the Case For Unit Testing
In six weeks, two verification engineers discovered 10 defects in the UVM-1.1d release. The defects were not lurking in the shadows of complex corner scenarios; it didn’t take hours of constrained-random stimulus to discover them; nor was there any need for functional coverage with intelligent feedback. These were simple, obvious defects discovered with an equally simple and obvious technique. It’s called unit testing and it comes from software development where the value placed on code quality and maintainability is higher than what we aspire to in hardware.
UVM-UTest is an open-source project with the goal of demonstrating how unit testing can be used to bring software level quality and maintainability to design and testbench code. This paper documents the development of UVM-UTest, how it was conceived, statistics and style for unit tests written, its success in terms of defects found in UVM-1.1d, the lessons learned and recommendations for applying unit testing beyond the UVM. The paper concludes by introducing a new UVM component, the uvm_boat_anchor, that further demonstrates the perils of continuing to ignore the value of this basic yet entirely necessary technique.
I’ve submitted proposals twice in the last few years. Neither was accepted. Agile development and test-driven development – the topics I’ve submitted – aren’t your average mainstream, audience grabbing topics so that wasn’t overly surprising. Maybe this proposal is different. Or maybe it’s not. Either way, I’m leaving it for you to decide.
If there’s enough interest, I’ll submit it and see what happens. If you really like the abstract, share it with your colleagues so they can vote, too. Or if you hate it, may as well share it, anyway. I don’t mind people seeing my incredibly terrible ideas. I’ll just entertain myself with it.
Proposals are due Aug 27th so share the link and get your votes in asap!
The point of the open-source UVM-UTest project we’ve been working on is to demonstrate how unit tests can be used to lock down the functionality of legacy code. Being able to run the unit tests means that when you’re changing code you can verify existing features won’t break during maintenance. We think that’s valuable. A nice side benefit of the project is that we can also show unit testing as being effective for increasing the quality of code hardware engineers write. How can we say that? Because over a 5 week period we found 10 defects in the UVM 1.1d release. Continue reading
We did our first UVM-UTest challenge on Friday. It went surprisingly well so now we’re going out on a limb by inviting others to take the challenge.
The “rules” of the challenge are:
- A team of 3 has one hour to break our unit tests (we used a lunch hour so it was like a lunch-n-learn… except it wasn’t boring).
- If a team can make functional changes to the uvm-1.1d library that our tests do not detect, you win. If our unit tests detect the functional changes you make, the unit tests win. Pretty simple.
- We guide you through the challenge so you know what code/classes you’re able to change (we also keep notes for where we can improve our test suite). Continue reading
After less than a month of development, the open-source UVM-UTest project is ready for download by the masses. If you care about code quality and maintainability, you’ll be interested in what we have.
For instructions on how to download and run UVM-UTest, take a look at the UVM-UTest Getting Started page. If this is the first you’ve heard of UVM-UTest, you may also want to check out the original UVM-UTest announcement to see what we’re up to. In that announcement you’ll see links to the project charter and progress page. Also be sure to check out The Most Descriptive UVM Tests Ever Written. Those and others like it are part of today’s release.
The next step for us comes this Friday at Noon EST when we have our first UVM-UTest challenge. That’s where we challenge some of our colleagues to find holes in our unit test suite. If they can break the UVM library without our unit tests detecting it, they win. If our unit tests detect all the changes they make, we win. We’ll keep you posted on how that turns out!
..and of course… if you’re interested in taking the challenge, we’d be happy to hear from you :)!
If you want to know how the uvm_printer works, in detail, I don’t think you’ll find a better explanation anywhere than my unit tests. This comes after I went through a refactoring exercise in our open-source UVM-UTest project – that’s where we’re unit testing the code in the UVM framework – specifically for the purposes of making my code more readable.
Here’s one of the tests I’ve written to get the idea of where I’m going:
A colleague and I have just started a new open-source project that we think will demonstrate the merits of unit testing in hardware. It’s quite an ambitious project called UVM-UTest. The framework-under-test, in case it’s not obvious, is UVM.
The project is hosted on Github. If you’re interested in knowing more about what we’re doing, here’s a link to our UVM-UTest project charter. The entire project charter is one hand-written page so reading it won’t take long. It’ll give you a better idea of what we’re doing and why we’re doing it. If you’re interested in the abbreviated version, I’m comfortable paraphrasing it as:
We’re unit testing UVM.
We have a progress page that shows you how far we’ve come so far. It shows classes and class members that we’ve unit tested. We update that a few times a day so it’s always quite current.
We’re about 3 weeks in and the project is going well. We’ve got about 250 unit tests that take about 10sec to run and we filed our first issues in the eda.org mantis database last week (you can filter out the issues filed by ‘Neil_Johnson’ to see what they are).
There’s a first planned release of UVM-UTest scheduled for June 7. That’s when you’ll be able to download the code and run the tests we have for yourself!