Contents
ABSTRACT
This paper describes a project management approach called “Kanban” being used successfully in software development today. Using Kanban, a team’s complete process is drawn on a board. Queues are created on the board to indicate steps in the entire development flow. Tasks are added to the first queue and progress to the last queue as each step is completed.
The success of this technique is largely due to its simplicity, ability to visualize a team’s process, and adaptability to changing conditions. By visualizing the flow it allows the team to look at the process, identify bottlenecks and optimize the entire flow on a continuous basis – rather than at pre-ordained times defined by management.
We provide details for the Kanban process and illustrate how it can be effectively adapted into an IC development flow. We’ll show that Kanban is ideally suited to a multi-disciplinary team that ASIC/FPGA projects require.
Comments? Send to bryan.morris@agilesoc.com
Introduction
This paper provides background for a project management system called Kanban that originated in manufacturing as part of the “Lean” focus, and then has been successfully adapted for software development. In this paper and we provide some background and demonstrate that Kanban can be adapted for an IC development with equal success.
The key reasons behind adopting a Kanban approach in your IC development are:
- It provides a visual representation of your process flow – enabling people to more easily pick out where there are opportunities to optimize the work flow (i.e., in the Lean manufacturing terminology – eliminating “waste”).
- It quickly highlights issues and bottlenecks in the process so that management can apply the correct resources to resolve any issues as quickly as possible;
- It creates opportunities to ensure that the team is always working on a feature that has the highest customer value.
- It is easy to adopt to start as simply as possible, and then incrementally optimize your team’s or project’s processes.
This paper is broken in two sections: the first section describes the Kanban methodology; the second section provides a concrete example of one of many ways to use the Kanban approach in an IC development. Kanban compared to other Agile methods is a very low overhead and simple methodology with only two simple rules. The fact that it does not proscribe the steps you’ll take in your process is part of its attraction – it can be added into your existing flow very easily.
Kanban Background
Overview of Kanban Process
Kanban (from the Japanese for kan=”visual” and ban=”card or board”) is a process tool with origins in Lean manufacturing. Lean manufacturing is a technique to “coordinate the flow of parts within the supply system” [2]. Kanban was created by Taiichi Ohno and first used at Toyota. Applied to manufacturing, Ohno envisioned Kanban as a just-in-time (JIT) system where “parts would only be produced at each previous step to supply the immediate demand of the next step” [2]. By limiting output to meet the demand of the next step, work queues remain short and inventory levels are held low resulting in a levelled and continuous flow through the system.
While there may seem to be few similarities between a manufacturing process and IC development, the concepts of work queues, inventory and levelled flow are still applicable. In IC development, Kanban is more about coordinating the flow of work within the development system –- compared to coordinating the flow of parts through a supply system. While these two goals are fundamentally different, we see that there are similarities that we can leverage in our flow.
This paper explores the adaptation of Kanban in an IC development flow.
For more information on Kanban please see http://www.kanbandistilled.com/ for a concise description of how Kanban is used in software development. Anderson’s Kanban [7] also provides an excellent introduction and great suggestions on adopting Kanban.
Kanban In Action
Kanban itself is very simple. As its name implies, it involves visually mapping your workflow on a board. The visual aspect is an effective tool for managing work-in-progress (WIP), identifying process bottlenecks and achieving level flow [3][6]. The key benefits of Kanban are its simplicity, flexibility and most importantly, the associated visuals.
A Kanban board is normally a table with a series of workflow queues. Workflow queues are defined by a team and correspond to some meaningful stage of development. Work such as new product features, customer requests, or work orders flow across the Kanban board through each workflow queue until they are completed.
Figure 1 – Kanban Work Queues
Kanban is very flexible in terms of how and where it may be applied. There are no rules as to how many workflow queues are used, or the granularity of each queue. There is one proscriptive rule: the Kanban board MUST be visible and accessible at all times to every team member, and only two less proscriptive rules that must be adhered to:
- Limit Work In Progress (WIP).
- Optimize the Lead time.
In brief, the first rule “Limit Work In Progress” ensures that the resources available to a project are not over-committed. While the second rule “Optimize the Lead Time” looks to continuously optimize a project’s processes and resources by eliminating all waste in a system. These rules are explained in detail in sections Rule #1: Limiting WIP and Rule #2: Optmize Lead Time, respectively.
The number of workflow queues must be defined by the team. However, that work flow need not be every queue using a fine granularity. The team can start with three simple queues: Todo, In Progress, and Done. Then gradually add additional queues to better reflect the process the team creates and evolves. The number of queues is dynamic and in the team’s control. The team can now see the entire process, and so at any time identify and suggest workflow changes, deletions or additions to the number and meaning of each queue to optimize their work flow. Another recommendation is that initial workflow could represent the process that the team currently uses.
A Kanban board may by used to manage development flow development of a single product, multiple products, maintenance mode activities, etc. One hard requirement of Kanban is that a Kanban board is visible and available to the team at all times. While the tasks and queues an be captured electronically (e.g., a web based capturing system), Kanban boards have been found to be most effective in a low-tech form constructed on walls with tape grid and sticky notes or index cards in high traffic areas [3][6].
Kanban Board In Action
Figure 2 – Example Kanban Board
The Kanban board in the figure above captures a very simple example workflow. In this example, by the support team in an organization that develops and sells Electronic System Level (ESL) intellectual property (IP). The support team receives, develops and deploys IP upgrades at the request of its customers. Work items in the form of new feature requests for the maintenance team are asynchronously initiated by customers. Features are the items that flow across the Kanban board.
Continuing with the above example, the support team creates four workflow queue on the Kanban board:
- Design
- Implementation
- Customer deployment, and
- Done.
New feature requests are copied onto a physical feature card (e.g., a sticky note) go into a backlog queue (not shown). When the team chooses to work on a particular feature, it is pulled from the backlog queue into the Design queue. As the team designs, implements and deploys the feature its corresponding feature card flows across the Kanban board to eventually end up in the Done queue.
It is important at this step to define exactly how the items flow across the Kanban board because it is a seemingly simple concept that is critical to the success of using Kanban board. Work items can only be pulled from an upstream queue when a downstream queue has an available slot. In the example above, you can only pull from the Design queue when the Implementation queue has an available resource. The next section will illustrate this concept further.
Rule #1: Limiting WIP
There can be more than one feature in progress at a given time. At the time the snapshot in Example Kanban Flow for IC Development (above) is taken, the support team has two features being designed, three being implemented, one waiting to be deployed — for a total of six in progress — and six already deployed. However, there are always reasonable limits that need to be applied for any of these queues. These limits reflect the project’s limited amount of people or physical resources e.g., compute farm or license limitations. Any work that remains queued in a system is one source of waste in the workflow. How work queues manifest themselves as waste is due to rework, stale requirements or overflowing bug tracking databases, etc.
These limits affect how much work can each queue can realistically accept at any given time. If there are three architects/designers that can accept work destined for the Design queue then a reasonable limit to that queue is three. This highlights the first rule in Kanban (#1 Limit Work in Progress (WIP)). Effectively it applies a queue depth where work (or the number of features captured on a sticky note) in a given workflow queue cannot exceed the queue’s WIP limit. Once a WIP limit is reached in a given workflow queue, the team must advance work items on to the next queue in the workflow before others can pulled onto the current queue. To illustrate, if a WIP limit of three is set for each workflow queue used in the previous example (shown below), and then assume the one or more of the features in the Design queue are complete, while all three features in the Implementation queue are not. In this case, the design team cannot pull another feature from the backlog as their WIP limit has been hit. This demonstrates where a downstream queues applies backpressure – effectively blocking work from starting in upstream queues. In other words, the owner of the work-queue is pulling work from the previous queue when they are ready to accept the work. This is a different model than many current work environments where work is sometimes dumped onto the next person in the flow and they are clearly not ready for it.
Figure 3 – Simple Workflow with Work In Progress Limits
At this point the team determines why the features in the Implementation queue are taking longer than expected and take appropriate action. These solutions can include any of the following:
- shift resources (e.g., people or compute resources) assigned to the Design queue or from elsewhere to the Implementation queue to help them to alleviate the bottleneck;
- simplify the task;
- allow the resources assigned to the Design queue to multi-task – effectively transiently increasing their WIP — and start another task from the backlog queue (Admittedly, this breaks the Kanban Limit WIP rule, but it is a pragmatic solution);
- allow the resources assigned to the Implementation queue to park the feature for a later date;
- or any other method to alleviate the bottleneck in the Implementation queue.
A clear benefit of ensuring that a Kanban board is visible at all times to everyone on the team is that when bottlenecks occur they are highlighted to the entire team very quickly. Then the team can decide on the most appropriate action to resolve the issue. This creates a shared responsibility with the team to ensure that the team’s flow is optimal. Regardless of the chosen action, when at least one feature is implemented and moved from the Implementation queue to the Deployment queue, another feature may be pulled from the Design queue into the Implementation queue, and then finally the Design queue can pull a new feature from the backlog.
Rule #2: Optmize Lead Time
The second Kanban rule “Optimize Lead Time” ensures that your team is continuously looking for opportunities to optimizing how they work, or the process itself i.e., to eliminate waste. First a definition of “lead time” and “cycle time”. As shown in the Figure 4 – Lead Time vs. Cycle Time below:
- lead time is the time elapsed between when a feature is first considered (i.e., added into backlog queue) and when it is finally complete and added into the Done queue;
- cycle time is the time elapsed between when a feature is pulled from the backlog queue and when it is finally complete and added into the Done queue.
Figure 4 – Lead Time vs Cycle time
A distinction between the lead time and cycle time is that cycle time captures your team’s efficiency to implement and deliver a product.
The rule to Optimize Lead Time is intended to focus your team on the fact that your entire process should be continuously reviewed, looking for ways where your team can be made more effective and productive. Eliminating wasteful cycles in the process is a key element.
As Kanban visualization allows the entire team to see the complete process, this is a first step to allowing the team to optimize the process. As the team works on the project, there is always time to make suggestions for adding or removing steps from the process — with the Kanban board evolving as the process evolves. By having the board be highly visible, this creates both a mechanism and an opportunity to discuss the process.
Techniques a team can use to optimize the lead time are:
- breaking a step into multiple smaller steps;
- pulling together several steps into one;
- automation through scripting;
- adding additional resources e.g., compute infrastructure; and
- additional testing / review cycles to limit downstream waste activities.
Applying Kanban in IC Development
As the Kanban process is simple, adopting it as your team’s process for IC development is also straightforward. Recall that Kanban relies on two rules:
- Limit Work In Progress (WIP);
- Optimize the Lead time;
The suggestion of many Kanban thought leaders is to create the initial set of queue based on your current process and then evolve from that starting point [4]. At the very least, as you progress through your project you’ll be able to understand and create a clearer picture of where your team spends a great deal of its time, and understand where any optimizations will yield better results.
The following figure illustrates one example flow that could be applied to an IC development project.
Figure 5- Kanban Flow for IC Development
In this example we assume there are the following roles:
Product Manager
Responsible for collating customer or marketing input and prioritizing the feature requests that will be worked upon.
Engineering Manager
Responsible for ensuring that the engineering team is working effectively as possible. Their role is to clear roadblocks, ensure required resources are available for the engineering team, and ensure that the engineering team can focus on their task at hand.
Engineering Team
Consisting of the engineers responsible for implementation, validation and delivery of the system.
This can include any or all of the following: RTL designers; functional verification engineers; software engineers; DFT engineers; physical design engineers and emulation integration engineers.
The example flow and the associated activities for each queue are briefly described:
Backlog
This queue holds all the incoming features that need to be implemented.
This is the initial stage in the development process. The recommendation is that only Product Managers are allowed to add to this queue. However, if the Engineering Manager or Team wish to add a new feature (e.g., rework), it must be approved by the Product Manager.
In the example flow above the features are colour coded. This is purely optional, but is useful to prioritize the features to be worked upon. However, Kanban does not suggest any rules around colouring the features so the value of using colour and the associated meaning is completely at the team’s discretion.
Planning
This is the first queue where the Engineering Manager & Team pulls the highest priority feature from the Backlog. Its main goal is to fully understand the feature by asking the Product Manager enough questions to accurately gauge the amount of work required. It can also be considered a staging area to rationalize the incoming features.
The activities during this stage can include:
- Engineering Manager: Pulling the next highest value feature from the backlog queue.
- Engineering Team: Discussion with the Product Manager to ensure that the feature is completely understood. This discussion must include the Engineering Team to ensure complete understanding.
- Design, Verification: Create a first level design of the architecture from this discussion. This group should also consider the other elements currently in the Backlog queue as a roadmap for the architecture. That is, it may be easier to consider future work items when choosing one architecture or design pattern.
- Verification: Capture/update verification environment architecture in a Unified Modelling Language (UML) spec. This adds clarity with minimal overhead. The minimum suggested diagrams that should be created are a Use Case, Static Class and Sequence diagrams.
- ESL: Create an initial ESL Modelling Specification, describing the design for an ESL model that can (and ideally should) be created to perform architectural analysis.
- Verification: Verification Planning creation/updates.
Implementation
This queue is managed by the engineering team. They pull from the Planning queue the next highest priority feature. At this stage it is possible that a single feature is too large in scope for a single team. If so, the feature should probably be split into multiple sub-tasks to allow for parallel implementation of the feature.
Upon accepting a particular feature the entire Engineering Team should have an opportunity to discuss the feature with the Product Manager. Their goal at this point is to ensure that they understand exactly what needs to be designed and implemented. It is also expected that these kinds of discussions and clarifications regarding the feature happen continuously throughout the remaining stages of work.
The activities that can take place during this stage can be as follows:
- Can split the feature into multiple sub-tasks e.g.,:
- Design: design and create RTL. It must run cleanly through a linting tool, compile and synthesize;
- Verification: design and create a block level verification environment and a basic sanity test of the feature;
- Verification: creating functional coverage model and review with design engineers;
- Software: create the usage model by identifying the control plane aspects (e.g., register map) that needs to be created for this feature. Possibly create the initial driver loads.
- ESL Architects: update the ESL model to reflect any design or control plane changes.
- Emulation: preparing the system level environment for integration queue;
- Physical Design: creating scripts and environment for the physical design.
- Updates to Design spec, verification plan and ESL Modelling Specification. This would include updating the UML diagrams created during the Planning stage and adding some context to pull the elements together. The intent is to capture the initial view of the design in a minimal way.
It is expected that all sub-teams in this stage will discuss (and argue) during this stage on the best solution — especially around the usage model of the feature being developed.
Integration
The intent of this stage is to pull the next highest value feature from the Implementation queue and provide an opportunity to pull together sub-tasks comprising a single feature that:
- have the most basic and critical aspects implemented in both the RTL and verification environment,
- have the basic usage model worked out between hardware and software,
- have sanity metrics identified, etc.
In essence, have enough of the design complete to begin sanity testing. This is the first opportunity for the design, verification, software and emulation teams to work together to ensure that the feature has been implemented and verified to a sufficient level of detail.
The activities that can take place during this queue are as follows:
- Verification: Ensure a particular feature works in a sanity environment. This will require debug and implementation of additional capabilities into the RTL, verification and software usage model.
- Software: software drivers can begin to be developed as the usage model should be well identified.
- Emulation: If possible, running the design through some long-term sanity tests on the emulation platform.
- Design: Assist in debug and continue to develop the remaining aspects of this feature.
- Physical Design: Review the physical aspects of the design to date (e.g., the implemented clocking strategy, synthesis reports, etc.) to gauge the design as it matures.
The goal for this stage is to ensure that the design passes all sanity tests, and the RTL is complete from an implementation point of view — not necessarily fully tested.
Verification
At this stage, the intent is for the verification, software and emulation teams to be to “beat up” the feature to ensure that it works completely. That is, the goal at this stage is to ensure that the coverage is complete and the ideally software has done some co-verification.
The teams will take the next highest value task (or sub-tasks that can be logically grouped) from the Integration queue and the activities that can take place during this queue are as follows:
- Verification will continuously review results from regressions and incrementally add the stimulus and self-checking in the verification environment and test cases with the goal of completing coverage.
- Verification and Design work together to debug and resolve all issues identified in the constrained random regressions.
- Verification, Software: Run system level regressions and software co-verification.
- Emulation can also continue to develop additional long-term system level tests to validate the design.
- Physical Design can begin push the design through the initial synthesis tasks and work with design engineers to resolve any issues.
At the end of this stage the design is complete from a functional perspective. It should be ready to be pushed through the physical design flow to ensure its physical correctness.
Physical
The Physical Design team pulls the next feature from Verification queue and works it through the physical design process. As constrained random regressions will continue throughout this process it is possible that new issues will be debug and resolved. If these occur, they should ideally be handled as Engineering Change Orders (ECOs) at this point, and managed accordingly.
The activities that can take place during this queue are as follows:
- Physical Design: Runs through complete PD process.
- Verification: Constrained Random tests will continue to be developed, added to the regression and analyzed.
- Emulation: System level tests and emulation will continue.
- Design will resolve issues and submit issues via an established ECO process.
- Software can now complete their development of their drivers.
The goal for this stage is to complete the physical design process and get the RTL to the point where it is ready to be taped-out.
It should be made very clear at this point that the “RTL” referred to above is only the RTL for the feature that the Design, Verification, Emulation, ESL, Physical Design, and Software teams have completed as a team. It is sub-set of all the features that are to be included in the final tapeout. So this is an iterative process where each feature is incrementally added to the chip until the next queue.
Ready for Tapeout
This queue is intended to be a staging area for all the features in a system that is to be taped-out. The only activity expected at this point is approval from all engineering teams that the feature is complete and ready to be shipped. Ultimately, the Engineering Manager is responsible for declaring which features are “complete”.
Release
This queue is intended to be managed by the Product Manager to decide when a set of features can be released to manufacturing. The goal of this queue is to pull from the Ready for Tapeout the set of features that comprise the marketable product.
As the feature has been designed and verified at the block and system level and pushed through the physical design process it should be complete. Clearly care should be taken if during this stage a particular feature is NOT selected to be part of the release as another included feature could have a dependency on the excluded feature.
We have represented one possible flow through a Kanban there are several other approaches that could work including creating a set of horizontal ‘swim-lanes’ that would differentiate the work across the multiple teams. As shown in Figure 6 – Kanban Flow for IC Development with Swim Lanes.
The WIP limits is the sum across all the teams. The horizontal swim lanes indicate which functional group has the current responsibility. It is feasible that the the feature will bounce back and forth between functional groups e.g., when a feature is in the Implementation queue:
- design completes the initial implementation;
- verification pulls it into their queue and applies the initial set of verification tests against and identifies several bugs against the design;
- design pulls it back into their queue to correct the bugs;
- verification pulls it into their queue to re-verify;
- continue as required.
Figure 6 – Kanban IC Development with Swim Lanes
Other Considerations When Introducing Kanban
This section provides some guidance on other aspects of using Kanban in an IC flow such as what kind of information can be contained on the Kanban card; the mechanics of moving a card from one queue to the next (even backwards to the previous queues), etc.
As stated previously, the most effective Kanban board is one that is visible by all team members and can be easily updated and recreated, if necessary. Many teams start out with a very large whiteboard with colored sticky notes or a bulletin board with index cards and push pins. Once the team is comfortable with the process you can evolve your team to an electronic Kanban board [5]. Then create vertical columns defining the start and end of each queue in the workflow with the workflow queue name at the top of each column in the order that they usually follow one another.
The information that is contained on the sticky note or index card can include as little or as much information as your team thinks they’ll need. Examples of information that a Kanban card can hold are:
- Unique Identifier: a short feature name and/or number– allowing the team to identify the feature quickly;
- Owner: the name or initials of who is currently responsible for completing the work for this feature in the current queue;
- Dates: When the feature was started; when it was added to the current queue; when it was last updated; what is the estimated effort/completion date; actual efforts/completion date, etc.
- Priority: Importance of the card. The use of colored cards can also visually convey this information.
- Issue ticket number: Providing a history for this feature through an issue tracking system.
- Blocked: Indicating that this feature has blocked.
- Exit Criteria: Defining what needs to be complete before this card is considered “complete” for the current workflow queue.
The amount of information you can add to the card is up to your team. When first adopting Kanban, use as little information as possible and then add information only when someone on the team says “I wish we knew X” when you are discussing the cards.
As suggested earlier, when creating your workflow it is probably best to start with the process you are currently using. At this point your team should also be honest with itself and put up the process you are actually using rather than the one you are supposed to be, or would like to be using. Once you have an honest starting pointing you can start working on optimizing your process to what truly works best for your team.
When moving a card from one queue to the next it is important that the person who is pulling the card into the next queue understands that it is complete and ready. This includes talking to the owner of this card for the current queue to get an understanding of what has been done. The new owner in the subsequent queue should update the card with any needed information (e.g., the date when it was pulled into the queue), and then simply moves the card into the next queues correct column on the Kanban board. If it is discovered that the feature was not, in fact, ready to be pulled to the next queue, it should be discussed with the previous owner, and the card’s information should be updated again and the card moved back into the previous queue.
To break a feature down into a series of tasks there is a flow described in [6] that is effective:
- Feature: A Feature is a behaviour of the system that is described in terms that is meaningful to a user of the system, or a customer
- Story: A feature is broken down into a set of “stories” that describe how the feature behaves under different events or circumstances; it is also described briefly in terms that are meaningful to a user. Creating stories is discussed in [1].
- Task: A testable unit of a story that can be worked on; these are described in terms that are meaningful to an engineer. They are testable in that they have some objective measure indicating that it is right or wrong.
The best part of this process is the discussion that happens with your team as attempt to break the problem down. The discussion usually leads to both a deeper understanding of the specific feature, and broader understanding of the feature in the context of the entire project.
Summary
In summary, Kanban is a simple process tool with flexibility and power when it comes to managing the flow of work through a development system. While workflow queues and WIP limits are prescribed in Kanban, their exact details are left to the discretion (and discovery) of the team. The details of how each workflow queue is managed and resources used are also left to the team’s discretion. Finally, discussion on Kanban in this article is meant only to serve as an introduction. While Kanban itself is relatively straightforward, the motivations for using it and the context in which it was developed may require more explanation. For more on Kanban and how Kanban is applied in lean manufacturing and product development, refer to the further reading.
Conclusions
This paper provides some background on the Kanban process and why it has gained traction and been proven successful in the software development world. We show that Kanban is a simple and effective methodology that can flexibly manage the work flow for the development of a system. It provides the team with opportunities to work together and easily identify blocking issues and resolve these issues much faster than traditional project management techniques.
Kanban allows the team to define and refine their process as the project progresses. Identifying areas to optimize and improve the productivity of the team. While visual workflow queues and Work In Process (WIP) limits are proscribed in Kanban, the exact details regarding what each queue is responsible for, the kinds of activities that can take place during the queue, exit criteria for the queue, and what are the limits to how many features or tasks the queue will accept, are left completely to the discretion of the team. In fact, many teams start with one workflow and transform their process as the project proceeds into a slightly, or sometimes massively different work flow.
In the second section of the paper we outline an example Kanban workflow and the associated tasks that could be executed during the project — demonstrating one team’s solution to using Kanban in an IC development.
This paper demonstrates that Kanban can be used effectively in an IC development and be as successful as it has shown in the software development world. The authors recognize that the Kanban approach will not appeal to all teams – compared to the Scrum technique (as discussed in the authors’ previous paper entitled “Agile IC Development with Scrum” [1]).
Kanban is an approach that favours starting from where you are and the optimizing only when necessary. Using Kanban provides a team with the ability to see their entire process flow, identify and resolve where there is “waste” in the system; identify and resolve issues quickly, and ensure that features are developed in a priority order that reflects a customer’s (or product management’s) highest values. There are elements here that should appeal to many teams. The cost of starting a Kanban approach is minimal; the possible payback in productivity, quality and team morale can be quite high.
References
These articles and books provide a good background into the Agile methodology in general, and the Kanban methodology specifically. We highly recommend the reading of Anderson’s [7] for a thorough understanding of Kanban when applied to a non-manufacturing environment.
- “Agile IC Development with Scrum”, B. Morris, N. Johnson, SNUG 2010 San Jose
- “Lean Thinking”: Womack, Jones, Roos 1990
- “Kanban vs Scrum”, Kniberg, Skarin 2010, http://www.crisp.se/henrik.kniberg/Kanban-vs-Scrum.pdf
- David Anderson, „Kanban System for Software Engineering“ Video Presentation http://www.infoq.com/presentations/kanban-for-software
- LeanKit Kanban, http://leankitkanban.com/Home. But there area many,many more electronic Kanban systems.
- “Visualizing Agile Projects Using Kanban Boards“, Kenji Heranbe, http://www.infoq.com/articles/agile-kanban-boards
- “Kanban, Successful Evolutionary Change For Your Technology Business“, David J. Anderson
Comments? Send to bryan.morris@agilesoc.com