We’ve looked at why teams should consider doing TDD; we’ve looked at how the roles and responsibilities change between design and verification experts; now let’s look at when everything happens. This is the key to seeing how everything fits together.
Most teams nowadays chose to split their verification effort up as a combination of block level and top level testing. Block level testing is usually applied to every subsystem in a design and intended to exhaustively cover all the features of each subsystem. Once the block level testing is done, the exhaustively tested pieces are integrated and tested in a top level testbench.
The need for block level testing is justified by the fact that the arms-length control and visibility normally available in the top level testbench is inadequate or inconvenient for testing the first and second order functionality we talked about in the last post. The need for the top level testing is obvious as well because the block level testing does nothing to verify the interdependencies between subsystems and the operation of the system as a whole.
Makes sense… block level testing to verify the details; top-level testing to verify the integration. Here’s a picture of that as an timeline with the design, block level and top-level testing.
The designers start writing RTL. Verification engineers start building a block level testbench at the same time (or shortly thereafter as I’ve shown here). When the RTL and the block level testbench are done, the verification team will start running tests and collecting coverage results. When the block level testing is done, the block level design is signed off and it’s ready for top level testing.
The top level testbench, in the meantime, is usually started after the block level design and verification is well underway. The top level tests get run when most the block level designs near sign off. At the conclusion of top-level testing, the entire design is signed off and the design goes to the fab (or the FPGA goes to a customer, or the IP goes to another team or whatever). You’re done.
Now let’s look at the problems that normally pop-up and the reasons for why things never seem to work out as planned.
- The RTL is NOT done: I’ve harped on this before so I’m not going to spend much time on it here other than to say that I would challenge anyone that says the RTL Done milestone we all race toward is at all meaningful. You may have a bunch of code, but it’s not tested and it’s not done (see By Example: Done vs. DONE).
- The testbench is NOT done: same comment as above. Testbenches aren’t DONE when we say they’re done either. That goes for block level and top level (again… see By Example: Done vs. DONE).
- You can’t plan your way through integration testing: here’s the big one and the real reason you’re here today. I really believe I’ve seen top level testing planned as good as anyone could reasonably plan. Well partitioned design: check. Well defined block level interfaces: check. Standardized model interfaces: check. Uniform, reusable block level environments: check. Customer initiated use cases: check. Skilled team: check. The ducks were all lined up and still there were the same old issues that tend to blow apart a schedule. A plan is one thing; practice is usually something else.
Take those and other issues into account and what you get in practice is more likely to resemble this…
RTL done and testbench done mean you have a bunch of buggy design and testbench code that doesn’t really work. After saying they’re done, experts from both domains transition into a firefighting role as soon as the first few tests uncover major quality issues (the firefighting is shown as red extensions to the original estimates). While that’s going on, the person in charge of the top level testing finds out that everything doesn’t actually just fit together as planned. Because people are firefighting at the block level and because the block level is given priority, the top level people have to make due until help finally arrives; if it arrives. A lack of help means the top level testing drags on, eventually stumbling across the finish line days, weeks or months later than expected. By the time the design is finally released, people are exhausted from weeks of stress and sleep deprivation.
I used to be of the opinion that teams could plan their way through top level verification. Now I’m sure they can’t.
I think the difference between planned and actual release dates is caused by false confidence and ambiguity in our development process. I think we can overcome both with a new paradigm for hardware development that emphasizes early validation and prioritizes top level development. Here’s a simple diagram of this new paradigm…
The first thing to point out is the alternating test-design cycles at the unit level. This is TDD. That’s meant to clean up localized/unit level bugs in the design before they’re committed to the database. Notice also that the unit level design starts with a purple bar. Test first!!
Next major point is that there is no delay to top level testing; it starts immediately. As portions of the design are unit tested, they are also verified at the top level. Top level tests are where we measure progress. When the top level tests pass, then – and only then – is the code considered DONE.
A third note here is reserved for the role of block level testing. In this new paradigm, block level testing is no longer automatic because TDD is used to prevent many of the first and second order bugs and integration testing happens much sooner. While there may still be a need for block level tests to, for example, isolate certain complex features that are hard to isolate at the top level, exhaustive testing of all blocks within a design would no longer be the norm.
We’ll still do block level testing but it’s no longer “on” by default.
What’s required to keep everything together in this new paradigm for functional verification? One word…
The top level effectively takes priority in the testing, which means all the development at the unit and block levels must be synchronized such that it feeds the top level effort successfully. Everyone needs to know what is going on and how there work supports the combined effort of top level testing. That takes teamwork, plain and simple.
Q. What problems do you see during integration testing? Does unit testing and/or early integration testing address those problems?