By: Neil Johnson – Principal Consultant, XtremeEDA
This is an article that presents a combination of ideas from the worlds of Electronic System Level (ESL) design and lean product development. While the title may look a bit odd to IC and ESL developers–kanban probably looks like a typo–top-down ESL design is hopefully enough to pique interest.
That said, the title is not a completely accurate description of the article. A less engaging though more accurate title would be: Modified V ESL Design with Kanban. Now that is ahead scratcher! What is a modified V? What is kanban? What strange thing is created when you mix them with ESL design?
In short, the modified V is a top-down approach to ESL design credited to Brian Bailey and Grant Martin from their book ESL Models and their Applications. Kanban is a process tool with origins in lean manufacturing that is used to “coordinate the flow of parts within the supply system” (Womack, Jones, Roos 1990).
By supplementing the modified V approach to ESL design with kanban, the goal of this article is to support the modified V development as superior to current best practices.
Classic V Development
Before getting to the modified V approach, it is important to understand the unmodified V–or classic V development approaches proposed in Bailey, Martin 2010–and its shortcomings.
The classic V approach describes a process of system decomposition and reconstruction that most ESL and IC developers are likely familiar with. In the design phase, a system is sliced into abstract layers where the structural and functional details at each layer become increasingly fine. Starting with the system specification, design focus filters down through the macro and micro architectural layers to the detailed implementation.
While design of the system follows a top-down approach in the class V, verification and validation proceed bottom up. Implementation details are first verified on relatively small modules. Verification continues up to confirm micro-architecture and architectural level intent. Finally, verification reaches the system level after which system validation is performed.
Bailey, Martin 2010 identifies two shortcomings of the classic V approach. First is that because verification begins at the lowest level of implementation, RTL must be ready prior to making any meaningful progress. This is not to say that verification engineers cannot be developing tests prior to the RTL being ready; because those tests cannot be run until the corresponding RTL is ready, however, RTL development and verification are effectively running open-loop with respect to each other so real verified deliverables are no closer to completion.
The second shortcoming with the classic V approach is that bottom-up integration issues tend to be difficult to foresee and overcome. Using currently accepted best practices, the ideal situation involves a team reusing module level testbenches to verify integration issues as they work toward the system perspective. While the IC development community has put considerable effort into enabling and promoting testbench reuse, it remains a difficult task for many teams to efficiently and successfully overcome. Oversights, rework and the schedule slips they lead to are common during integration. A complete overhaul of a block level testbench, while perhaps not as common, is also possible.
While these shortcomings identified by Bailey, Martin 2010 provide a valid argument for a more effective approach, perhaps a larger issue with the classic V is the fact that verification in the system context is commonly ignored until late in the development cycle. The danger here is that the system level perspective is also the customer perspective. In many cases, therefore, the customer effectively has little or no insight into what they are getting nor any means of objectively measuring progress until well into development. This black out period may even extend to the time when engineering samples are delivered and validation is well underway.
Modified V Development
In ESL Models and their Applications, Brian Bailey and Grant Martin propose a modified V approach as an alternative to the classic V. As noted in Bailey, Martin 2010, verification in the modified V approach follows what is effectively an inverse path relative to the classic V. The system is validated first. After system validation, the architecture, micro-architecture and implementation are all subsequently verified.
The modified V approach, depicted in Figure 2, is made possible through the creation of one or more ESL models. The model at each layer is designed to validate/verify characteristics that are applicable to that layer. It is possible that one model may be applied in more than one stage of verification. In doing so, however, it is required that additions to the model do not destroy the integrity of previously verified model functionality (Bailey, Martin 2010).
The major advantage of the modified V approach is that integration verification as it exists in the traditional approach is virtually eliminated because the architecture and interfacing is essentially verified before implementation even begins (Bailey, Martin 2010).
While eliminating integration issues is definitely advantageous, perhaps greater is that validating the system level specification–or customer perspective–is given highest priority. Further, the ESL model may be used to actually demonstrate the specification enabling an opportunity for the customer to judge first-hand how accurately the specification captures their vision for the system. Such a demonstration with a paying customer is the ultimate form of early validation.
The ideal modified V development offers obvious advantages over the classic V approach but it does so by making two assumptions. First, it is assumed that building an ESL model of the system can be done much faster than RTL can be developed. With current modeling techniques, proper planning and reasonable goals at each layer of abstraction, this is becoming an increasingly reasonable assumption.
The second assumption is that decisions made at a particular layer of abstraction can in fact be verified at that same layer with minimal or no feedback from subsequent layers. And while not stated explicitly in Bailey, Martin 2010, this assumption does infer that development follows a waterfall type progression where:
- Sub-system development starts after specification has been validated;
- Major block development starts after the architecture is verified; and
- Small block development starts after micro-architecture is verified.
This article contends that verifying assumptions at each layer of abstraction without feedback from subsequent layers may prove unsatisfactory. Bailey, Martin 2010 indirectly provides support for that contention in an unrelated discussion regarding multi-application SoC development, “a corollary to Murphy’s law predicts that a new scenario will come along that will invalidate all the assumptions made through static analysis…Thus further dynamic, simulation-based, analysis is needed.”
This corollary to Murphy’s law is by no means used to discount the validity of the top-down modified V approach to system development and validation/verification; far from it. It is, however, being used to illustrate the potential risks for teams that see the value of the modified V approach but also, naively, assume a waterfall progression.
Recognizing there is real value in the modified V approach, it would be beneficial to supplement it with opportunities for “further dynamic, simulation based analysis” and the feedback they provide. Kanban can be used to do just that.
Introduction to Kanban
As stated in the opening, kanban is a process tool with origins in lean manufacturing that is used to “coordinate the flow of parts within the supply system” (Womack, Jones, Roos 1990). Kanban was created by Taiichi Ohno and first used at Toyota. Applied to manufacturing, Ohno envisioned kanban as a just-in-time (JIT) system where “parts would only be produced at each previous step to supply the immediate demand of the next step” (Womack, Jones, Roos 1990). By limiting output to meet the demand of the next step, work queues remain short and inventory levels are held low resulting in leveled and continuous flow through the system.
While there may be few concrete similarities between manufacturing processes and IC development, the concepts of work queues, inventory and leveled flow still apply. In IC development, kanban is more about coordinating the flow of work within the development system–as opposed to coordinating parts through a supply system. While these two goals are fundamentally very different, on a conceptual level they are very similar.
Kanban itself is very simple that involves visually mapping the workflow. Used properly, it is a very effective tool for managing work-in-progress (WIP), identifying process bottlenecks and achieving level flow (Kniberg, Skarin 2010).
The strengths of kanban are found in its simplicity, flexibility and visuals. A kanban board is normally a table with a series of workflow states. Workflow states are identified by a team and correspond to some meaningful stage of development. Product features, customer requests, work orders or whatever the team chooses to classify work, flow across the kanban board through each workflow state until they are completed.
Kanban is very flexible in terms of how and where it may be applied. There are no rules as to how many workflow states are used or the granularity of each state. The number of states and what each represents is completely dependant on the team and the process the board is meant to represent. A kanban board may by used to manage flow for development of a single product, multiple products, maintenance mode activities, etc.
One hard requirement of kanban is that a kanban board is visible and available to the team at all times. And while they can be captured electronically, kanban boards tend to be most effective in a low-tech form constructed on walls with tape grid and sticky notes or recipe cards in high traffic areas (Kniberg, Skarin 2010).
The kanban board in Figure 4 captures a very simple workflow that may be used, for example, by the support team in an organization that develops and sells ESL intellectual property (IP). In this example, the support team receives, develops and deploys IP upgrades at the request of its customers. Work items in the form of new feature requests for the maintenance team are asynchronously initiated by customers. Features, therefore, are the items that flow across the kanban board.
In this example, the support team has identified four workflow states on the kanban board: design, implementation, customer deployment and done. When a feature request is received, the request goes into a backlog queue (not shown). When the team chooses to work on a particular feature, it is pulled from the backlog and placed in the design state. As that feature is designed, implemented, deployed and completed, its corresponding feature card flows across the kanban board to eventually land in the done state.
At the discretion of the team, there can be more than one feature in progress at a given time (limiting the number of features in progress at a given time is discussed later in this section). At the time the snapshot in Figure 4 is taken, it can be seen that the support team has two features being designed, three being implemented, one waiting to be deployed–for a total of six in progress–and six already deployed.
Kanban is a highly flexible process tool that allows many interpretations. Besides being visible and accessible, one other notable rule in kanban applies to the notion of work in progress (WIP). WIP limits are used to reasonably minimize the size of each workflow queue. Effectively, work in a given workflow state is not allowed to pile up and exceed the queue’s WIP limit. If the WIP limit is reached in a given workflow state, the team must advance work items on to the next state before others can be added.
To illustrate, if a WIP limit of three is set for each workflow state used in the previous example (repeated in Figure 5), the three features sitting in the implementation state are effectively blocking progress in the design state. The team may slow or stop design work in order to clear the accumulated implementation work or assign more resources to implementation.Regardless of the chosen action, when at least one feature is implemented and moved to the deployment state, design may resume unabated.
Using the kanban board and taking into account WIP limits, one could identify bottleneck situations and initiate corresponding actions. In Figure 6, for example, the WIP limit is set to four. With four items in both the design and implementation workflow states but none in the customer deployment, this suggests that the implementation workflow state may be a bottleneck in the development system. The cause of the bottleneck and how it may be cleared–by reassigning resources, simplifying design, or some other means–is a decision that of course is left to the team and beyond the scope of what kanban prescribes. The board though is useful nonetheless for identifying these types of bottleneck situations.
In summary, kanban is a simple process tool with a lot of flexibility and power when it comes to managing the flow of work through a development system. While workflow states and WIP limits are prescribed in kanban, their exact details are left to the discretion of the team. The details of how each workflow state is managed and resourced is also left to the team’s discretion.
Finally, discussion on kanban in this article is meant only to serve as an introduction. While kanban itself is relatively straightforward, the motivations for using it and the context in which it was developed may require more explanation. For more on kanban and how kanban is applied in lean manufacturing and product development, refer to the further reading.
Applying Kanban to the Modified V Approach to ESL Design
Before applying kanban to the modified V approach proposed in Bailey, Martin 2010, it is important to again highlight the potential and potential risks of the modified V approach.
This articles contends that early prioritization and validation of the customer perspective holds that greatest potential for the modified V approach. When combined with the possibility of eliminating many of the integration issues that exist in current best practices, as noted in Bailey, Martin 2010, the benefits of the modified V approach are quite clear.
One potential risk however–particularly when following a waterfall progression–is that there are no opportunities to confirm design assumptions between system design, sub-system design, major block design and smaller block design.
In this section, kanban will be used to enable feedback opportunities between these different layers by creating workflow states that correspond to the steps in the modified V approach. WIP limits will also be placed on each workflow state. The ultimate goal of limiting WIP is to limit the number of unconfirmed design assumptions in progress at a given time and limit the damages caused by invalidated assumptions.
If kanban were used to describe the modified V approach with a waterfall progression, the resulting kanban board would appear as shown in Figure 7. All the features of the system–a modest total of 18 features in this case–would flow through each workflow state as one large batch starting with the system design state, eventually traveling through to the verify implementation state and into the done state. There is no WIP limit so the amount of work queued in each state is effectively unbounded; all the features in the system may fit into the same workflow state.
The kanban board in Figure 7 represents the state of development relatively early in the project cycle. The focus at this point, as it should be, is validating the customer perspective. Enabled by an ESL model, features can be demonstrated amongst team members and even with customers. In doing so, valuable system level product knowledge is gained and with it, confidence in the development trajectory.
Hidden in Figure 7, however, is one implementation level assumption that has yet to be verified simply because the ESL model built for system validation does not allow for it. Fast forwarding to a time later in the project, we will see how invalidating this assumption in combination with large batch processing causes considerable re-work.
As the system reaches the smaller blocks design state, the team comes to the unfortunate realization that the previously noted assumption has been invalidated by a critical hardware latency violation. Ripple effects of this violation extend well into other parts of the system, including software, where four of 18 system features are adversely affected. One of those features requires major re-work at the architectural level while the other three are impacted to a lesser degree; re-work for these three features is confined to the micro architecture level and below.
As a result of the hardware latency violation, progress on the kanban board has been reshuffled as seen in Figure 9. There are two important points to make with respect to the product and the team’s progress. The good news in terms of the product is that the customer perspective is left in tact. While there is considerable re-work required, the team estimates it can recover to deliver a product that it demonstrated to its customer earlier in the project. The bad news is that a lot of the effort the team has put into more detailed modeling, software development and hardware development will have to be redone. So while their customer may be happy that the system will still perform as expected, delivery will in fact be a number of months later than expected.
Assuming that further static analysis would have been inadequate to avoid this situation, a change in approach may help mitigate the damages, by either turning a months long schedule slip into a weeks long schedule slip or avoiding a schedule slip altogether. Placing a WIP limit on each workflow state is one possible way to mitigate this damage.
Consider the kanban board in Figure 10. The same product with the same feature set is using a WIP limit of three in each workflow state. The same invalid assumption exists as feature A reaches the smaller blocks design state and the same latency violation is observed. The latency violation impacts the same features as were seen from Figure 9: A, H, K and L. And although the violation is no less serious, the damages are heavily mitigated relative to what was previously seen for a number of reasons.
To start, however, there is still re-work required for feature A where, just as before, further modeling and analysis at the micro architecture level is required. This re-work was unavoidable because it was not possible to verify implementation details until this point in its progression.
Highlighting the advantages of placing a WIP limit on each workflow state, it is seen that the re-work required on features K and L is limited by the fact that neither has progressed beyond the major blocks design state. Re-work for feature H is avoided altogether because it is still in the validate specification state at the time the latency violation was discovered; the architectural analysis, modeling and verification that required re-work before has not even happened yet so no time is lost to re-work.
This is an example of the feedback potential enabled by kanban and an illustration of how limiting WIP limits the number of unverified assumptions in progress at a given time. In this case, valuable feedback from one state has increased the confidence in two others. Put another way: quickly finding one mistake has let the team avoid three others.
Further consideration should also be given to the timing of the discovery. It can safely be assumed that the latency violation was found much sooner in development because of the WIP limits imposed by the team. As a result, the team has not only limited the amount of re-work required, it is also able to convey associated schedule slips to the customer far sooner. In a reasonable scenario, its customer may even be able to adjust dependencies, thereby absorbing any negative impact.
This is just one simple example scenario that may play out using kanban to manage the process of modified V development. While more complex scenarios are likely in real world ESL design, the purpose of following a modified V approach using kanban remains the same:
- give priority to validating the customer perspective in a top-down manner; and
- use WIP limits to minimize the number of unverified assumptions in progress at a given time.
A top-down approach to ESL design and verification provides a welcome change to current best practices. The modified V approach proposed in Bailey, Martin 2010 certainly has advantages over more traditional flows.
The addition of kanban provides an aspect of process management to the modified V approach thereby helping to guide teams through successful adoption. Kanban can help to minimize the design assumptions in progress at a given time, provide opportunities for feedback and limit re-work in an ESL development flow where the customer perspective is given highest priority.
Finally, kanban is only one of many process tools used in the world of lean product development and agile software development. There are many more tools and techniques with a long track record of success though in hardware development, their potential remains untapped. The time has come for us to discover the tools, techniques and great potential that exists beyond hardware development!
For more information on the ESL modeling approaches described in the article, ESL Models and their Applications is highly recommended; chapter 4 in particular is recommended.
For further information on kanban (and an alternative process tool called scrum), refer to Kanban and Scrum: Making the most of Both by Henrik Kniberg and Mattias Skarin (found online at http://www.infoq.com/minibooks/kanban-scrum-minibook).
Bailey, B., Martin, G., ESL Models and their Applications (1st edition), Springer, 2009.
Kniberg, H., Skarin M., Kanban and Scrum: Making the Most of Both, C4Media, 2010.
Morgan, James J., Liker, Jeffrey K., The Toyota Product Development System: Integrating People, Process and Technology,Productivity Press, 2006.
Womack, J., Jones, D., Roos, D., The Machine That Changed The World, Harper Perennial, 1991.
(c) Neil Johnson 2010 – All rights reserved