A handy bit of guidance that I’ve gleaned from books on lean product development comes via the recognition that unfinished work sitting in someone’s in queue, which in lean manufacturing lingo is called inventory, is waste in your development process. Lean software practitioners take things a step further to describe untested and/or unreleased code sitting on a file server as inventory. I reckon doing the same can benefit us in hardware development.
It makes sense to me that built up inventory can be responsible for poor productivity or quality. Why? Think about it… do you do a better job when the amount of work in your queue is manageable or overwhelming? Do you work better when someone shows up at your cube with 10 feature requests or 1? Do you work better under the weight of 25 outstanding bug reports or no bug reports?
I’ll take manageable over overwhelming any day so I can see minimizing the amount of untested and/or unreleased code… I mean inventory… in your development process over time is a good thing. Minimize inventory and you’re minimizing waste. Minimize waste and you’re maximizing productivity and quality.
How about an example that applies to hardware development.
Think about the work flows throught RTL design. Designers are writing new code, reacting to bugs filed by the verification team and reacting to bugs filed by the implementation team. That’s a combination of new features (the green files in the pictures) and bug reports (the red files in the pictures) being added to their in queue while they push new/revised RTL to their out queue. As long as a designer can handle the work in his/her in queue without getting bogged down, everything is fine… which is usually the case near the start of the project when there’s only new features and not much for bugs. Effectively, a designer has 1 input stream and one output stream where he/she can manage the flow without interruption; new features come from a spec, new RTL passes smoothly to verification and implementation. Everyone has something to work on and things are manageable.
Now pile on a few bugs from the verification and implementation teams and watch system inventory start to grow. With 3 steady streams feeding their input queue (1 for new features and 2 for bug reports), the designer is suddenly overwhelmed and his/her inventory grows. While productivity might be the same – which I’ve actually learned is highly unlikely as a person is loaded beyond 80% capacity – it’s clear the designer has become a bottleneck. The implementation and verification teams are now waiting for new code so they’re not happy either.
So what do you do here?
- Keep piling up the inventory? I’d hope not. Your designer is going to burn-out. Quality and productivity are going to suffer.
- Add a designer? Possibly. You need increased design bandwidth and adding a designer is an obvious way to do that, but do you have a designer to spare? If not, then you’re going to have to get a little more creative than simply throwing more people at the problem.
- Train implementation and/or verification engineers to clear design inventory? Yes! The designer has become a process bottleneck. You don’t have anyone else to chuck into the fire. Implementation and verification engineers are already assigned to the project. They know the product and are working at a reduced throughput. Why not have them help clear design inventory?
I like the last option best, which coincidentally is found in lean manufacturing. Pretend you have 2 guys on a line, Joe passing an assembly to Jim. On Tuesday, Joe is moving a little faster so the line is starting to back-up at Jim’s station. Jim’s feeling the heat so instead of continuing to pile on, Joe steps over, gives Jim a hand for a few minutes, then goes back to his own station. Jim is relieved and everything goes back to normal. Next day, Jim is moving a little faster. Instead of sitting and waiting for Joe, Jim takes a few steps over and gives Joe a hand clearing some of his inventory, then steps back to his own station.
Pretty basic teamwork, no?
Now let’s imagine the same technique used in hardware development. In my example where the designer’s inventory is building up, implementation or verification engineers can help clear it instead of continuing to allow it to grow. Depending on the skill-set of the individual, the extra help could come in the form of some exploratory bug diagnoses, full-on coding of new features or anything in between. Obviously this does require implementation and/or verification engineers to step out of their area of expertise temporarily, but they’re doing so to improve overall process throughput.
It all comes down to working toward the good of the team instead of letting one person toil while 2 others wait around.
So you’re telling me to turn my implementation and verification experts into design experts?
I knew you’d ask that question! No… I’m not suggesting that at all. I’m pointing out that with minimal cross-training and a little hand holding, you could easily teach implementation and verification engineers to diagnose and/or patch RTL to burn off a little inventory in your development process. Hardware developers are smart responsible people, so to say it’s not doable seems odd to me. From experience in other industries, that kind of basic teamwork can improve quality and productivity so why not give it a try?