There are times when I try and convince myself that unit testing my verification IP may not be necessary. Believe it or not, I can put together a pretty convincing argument. On a good day I shake it off and do the right thing. Other times I’m just convincing enough. I know it won’t end well, but I skip the unit tests and go straight to the code.
Here’s a couple arguments I’ve used on myself to get into trouble.
I Probably Don’t Need to Understand this Legacy Code
It may not be the obvious place to start, but I happen to think legacy code is a great entry point when it comes to unit testing. If you’re new to it, unit testing legacy code can help you understand what you’re working with. You can start small by picking out a specific function, read the code to get a feel for what it does, then write and run some unit tests to validate your understanding.
To modify legacy code, I write unit tests around the portion I’m expecting to change. That locks down the legacy behaviour. From there, if there’s an existing feature that needs to change I modify a unit test to capture the change then update the code to match the new test. I try and make my changes 1 test at a time to transform the code a baby step at a time. Similarly, if there’s a new feature I write a new test then the code to match, 1 test at a time.
To me, that’s been the right way. But I’ve also done this the wrong way…
The wrong way starts with the idea that any changes I make are going to be small enough that I can fake an understanding of the legacy code. But the first small change may not quite be enough; maybe I just need to tweak this over here, another minor tweak over there. Add the small changes up and it’s not long before I have code that neither I nor the person that originally produced the legacy code understand. People keep finding little problems; I keep patching those problems. Sometimes patches turn into the right behaviour, but most often they don’t. When they don’t, I turn back the clock to somewhere near the beginning, I write unit tests around the original code I changed, update tests, update the code, etc. Basically, I go through all the steps I should’ve gone through in the first place but only after I get it all wrong, wasting my time and the time of others in the process.
This Code is Too Simple to Test
There’s the code that’s too big or complicated to want to understand with unit tests. Then there’s the opposite problem: I’m about to write code that’s too simple to test. I find this situation more tempting than the legacy code situation because I feel like I’m in control of the unknowns. I forego TDD (my usual tactic) and just spit up the code. If it compiles it probably works, then I move on.
Note that the too-simple-to-test problem can be some new standalone chunk of code. It can also be a simple feature within a more complex component (this is more common for me); I use TDD for most of it but skip a few tests here and there when I think they’re not necessary.
But they’re usually necessary.
For me, skipping unit tests is a careless way to write code. Even if I get it right 9 times out of 10, the 10th time can lead to enough wasted debug effort to more than erase the time I saved by avoiding unit tests in the first place. Maybe some of the simple, tedious unit tests don’t add a whole lot. But some of them add way more than you think and it’s hard to tell the difference as you’re writing them. Safe thing to do is to take your time and write the tests.
There are other arguments to avoid unit tests but these are the two that I still seem to fall for. Simple new code, complex legacy code, it doesn’t seem to matter to me: I can make mistakes just about anywhere! As hard as I try, I can’t avoid unit tests. Sooner or later they all seem to get written anyway.
-neil
I’m also a big supporter of unit testing, but lately I’ve seen some situations where more unit testing wouldn’t have brought much benefit. This is mostly regarding collaboration between different classes.
An example: say I’ve got some block that takes data from a North bus, encrypts it and sends it out on a South bus. The North/South busses use different types. Encryption depends on a lot of other factors (configured key, design state, etc.). Since its behavior isn’t trivial, the encryption class is tested separately. When writing a class to model the entire transformation form North to South I would stub encryption out and just focus on the code that transforms from the North protocol to the South protocol. This code would also make calls to the encryption class to get the data that is supposed to pass:
class data_transform;
// …
north_item = encryption.encrypt(north_item);
south_item = convert(north_item);
What I’d be most interested in is that the ‘convert(…)’ method works properly. Since I stubbed encryption to not touch the ‘north_item’, the ‘data_transform’ class is easier to test, because I can set up my expectations much easier. At the same time, a requirement of the ‘transform’ class is that it also encrypts the north data. To test this would be rather difficult, because I would need to check that ‘encrypt(…)’ is called on the ‘encryption’ object with the appropriate arguments. I would also need to check that the result of this method call is used further on. All of this is pretty difficult to test and requires a lot of code to properly mock out the encryption class. I would say that the benefit is also limited, since forgetting to plug in encryption is something that would pop up immediately when running the code with the DUT. I wouldn’t spend too much effort just to test that the one line with the ‘encrypt(…)’ call is actually there, because this would require a ridiculous amount of testing code. Once we have Mockito style libraries that can automate this, I agree that this could be easily tested as well. That’s if we can ever libraries like Mockito in SystemVerilog.
The bottom line is that I wouldn’t blindly try to unit-test everything, but I’d try to figure out what interactions could be left untested. The key here is that I’ve only excluded the interaction. I would still thoroughly test the ‘convert(…)’ method, by making sure that I take all branches, etc.