Over the past 15 years, I have observed that managing dependencies has been a constant source of anxiety and a huge challenge for organizations adopting Agile methods. Meanwhile, dependency management has largely been a non-issue for organizations using the Kanban Method.
One of the most important, anti-Agile articles I´ve written is titled, The Tyranny of the Ever-decreasing Timebox (published in 2014, based on a speech I gave in 2009). This article “The Tyranny of Timebox Revised” is it`s new updated and revised version. Over the past 15 years, I have observed that managing dependencies has been a constant source of anxiety and a huge challenge for organizations adopting Agile methods. Meanwhile, over the same time period, dependency management has largely been a non-issue for organizations using the Kanban Method. This week we release our general solution for dependency management using Kanban, and in a series of articles, starting with this one, we look at why it is so much easier to manage dependencies with Kanban than with Agile methods based on time-boxed increments (sprints) such as Scrum and SAFe. This article revisits, revises and extends the original text from 2014, updating it to include the topic of dependencies.
Timeboxes
Agile software development methods, with the odd long since forgotten exception¹, adopt the use of fixed time increments, often wrongly called “iterations”². In Scrum, these are known as Sprints. A Sprint is a fixed period of time with a defined scope and a commitment³ to complete that scope within that time window. Originally, Scrum defined 4-week sprints. This was changed later, circa 2004, to a recommendation for 2-week sprints as the default. In general, it is recognized that agility is related to the frequency of interaction with the customers or business owners, and the frequency of deliveries. Hence, smaller timeboxes are more desirable, if you wish to pursue greater agility.
Batches
Software quality is often related to batch size and the time taken to complete a piece of work. The relationship is known to be non-linear, i.e. the defect rate rises faster and accelerates away as the batch size or length of time extends. Hence, smaller batches, completed in short periods of time lead to dramatically fewer defects. So, in theory, small batches of work are desirable.
Two ways to constrain small batches of work – timeboxes or WIP constraints
There are two ways to constrain the batch size of work: constraining the amount of time available to do the work, resulting in scoping to small numbers of requests that can be completed in the given time; or, simply constraining the number of work items, constraining the size of the batch of requests, also known as WIP constraints. All mainstream Agile methods adopt the use of timeboxes to constrain batch size by scoping work to fit in the available time. The Kanban Method instead adopts WIP constraints to directly constrain batch size.
At a small scale and with lower maturity organizations, it doesn´t much matter which of these two strategies you pick – they are both effective. Scrum is a perfectly good methodology to take a team from maturity level 0 to 1. However, as you both scale your ambition and the size of the organization, and improve the organizational maturity, with the goal of consistently delivering customer demand, within customer expectations, at larger scales (a product unit of 150 people, to a business unit of 600 to 1500 people or more), the timebox strategy breaks down. You cannot scale agility and organizational maturity with team-level timeboxes. The remainder of this article explains why not…
The Pressure for Shorter Timeboxes
As a result of all the advantages of smaller batches – higher quality, more frequent interaction between customers and the delivery organization, and the potential gains from earlier delivery of valuable, working software (also known as the avoidance of opportunity cost of delay), organizations have been under pressure to pursue shorter and shorter sprint lengths.
The Challenges of Shorter Timeboxes On the face of it, smaller timeboxes and hence smaller batch sizes for the sprint backlog, are a good thing. However, smaller timeboxes create three types of pressure that are often difficult to cope with and adjust to: firstly, smaller batches require an ever more detailed approach to requirements analysis and development – the need to write ever more fine-grained story which can be completed within the smaller time window; secondly, the need for an ever more accurate approach to estimation, so that a realistic commitment can be made; and finally, the need to know whether a piece of work will be affected by dependencies, and if so, a need to track and manage such dependencies, between teams and multiple sprint backlogs, and potentially, across sprint boundaries. A failure at any of these three things causes the timebox approach to break down and fail spectacularly. The timebox approach is inherently fragile on a large scale. Let´s examine why…
Requirements Analysis
Developing capability with a new and hopefully better requirements analysis technique, designed to provide the granularity to ensure work items are sufficiently small as to fit neatly into a short timebox, is hard, even when you know with confidence which technique you should adopt. The challenge with very short sprints is that there is, still after 20 years of Agile, little or no solid guidance on writing fine-grained, consistently small user stories. Even if there is a good method that your team is comfortable using, then (a) you now have a lot of analysis upfront, before sprint planning, and (b) you have introduced a peer-to-peer, or parent-child dependency management problem, that becomes especially acute as soon as you are unable to select all of the peers for the same sprint, with the same team.
Fine-grained requirements analysis coupled to short timeboxed sprints introduces a dependency management problem to Agile methodologies.
Alternatively, in the absence of strong guidance on requirements analysis, anxiety leads to undesirable effects such as breaking stories up into functional units based on information discovery activities such as “architecture story”, “design story”, “development story,” “test story” where the real customer value may now span across multiple sprints and peer-to-peer dependencies between stories across sprints is now a tracking requirement. This type of breakdown defeats the purpose of the ever-smaller timeboxes and creates a false sense of agility when in fact, customer value and quality are not improving, perhaps the opposite may even be true. A failure to instrument for and track customer lead time disguises the problem, and Agile teams focused locally and only on themselves, merrily ignore that they are not delivering customer value.
The concept of a ¨Design Sprint¨ (or other similar, upstream, pre-commitment information discovery) is now a recognized thing and you can take classes in it. Be under no illusion, design sprints are an anti`-pattern, they are an indicator that timeboxed increments in Agile methods are a dead-end – they do not scale! To be Agile you need to be small – small teams, small products, small codebase. To scale, you need to abandon agility and use much larger timeboxes.
Estimation
As sprints become shorter and shorter, the challenge of knowing whether a work item can be completed within the time limit becomes more and more important. Consequently, over the last 10-15 years, I have watched on the sidelines as the Agile community has pursued ever more elaborate means of estimation. The shorter the timebox, the more upfront effort is required to estimate whether the work will fit into the available time. Big estimation up front, is an anti-pattern and destroys the concept of agility – you need to estimate everything in your backlog first to play some magical game of Tetris trying to fit odd-shaped, fine-grained requirements into the sprint backlog. As we shrink the timebox, we increase the economic costs (the transaction costs of sprint planning) and decrease the efficiency of sprints. It is neither Agile or Lean.
Dependencies
What if a story in our sprint backlog gets blocked because of a dependency? That might prevent it from completing on time. And hence, the shorter the sprint timebox, the greater the need to identify dependencies upfront. This requires yet more analysis of as yet uncommitted items in the product backlog. And what if there are dependencies? Agile methodologies resort to organization design to solve this – the magical cross-functional team. The idea that everyone on the team can do everything that might be required to produce a top-quality product. For any firm that aspires to market leadership, this has never been true – you don´t win at anything with generalists, winning requires specialists. A small band of elite generalists can and does happen but again it doesn´t scale. You don´t staff a 500 person IT department, or a 600-person business unit with elite generalists.
All too often we see organizations, like one of those in a case study we use daily in our training classes – they had 5 development teams, 3 data-base administrators, and 2 user-experience designers. So either, you put the two UX guys in each of 2 teams, and the 3 database guys in the other 3 teams and now you have 2 front-end teams and 3 back-end teams, and you try to plan the sprints such that those teams only pick front-end or back-end stories, and oh whoops, you are disconnected from the customer requested functionality and have created dependencies, or you accept the truth and you use the dba´s and the UX people as shared services, and once again you have dependencies.
Dependencies are a fact of life in all but very small-scale software development, or mediocre development done by largely dilettante generalists. If you are doing anything at scale, or anything that aspires to market leadership then you have specialists, and you have dependencies. The Agile movement has been in denial of this basic truth – to be Agile you have to aspire to be mediocre!
Scaling, Business-agility and Timeboxes
Scaled Agile Framework (SAFe) approach to dependencies, is to reverse agility and instead use 3-month timeboxes called product increments. Effectively rolling the clock back to 1994, a big planning up-front approach, involving big analysis and big estimation up front – and lots of red string – to produce incredibly brittle plans that are unlikely to survive the first few of the 13 weeks intended. Scaled Agile Framework and its product increments and PI Planning quite simply aren´t Agile. To cope with the challenges and anxieties of agilists everywhere, it simply gives them permission to stop being Agile and do everything in quarterly time-boxes. It is an elegant solution but it isn´t Agile, rather it is the antithesis of Agile and the antithesis of the motivation for the lightweight methods movement of the late 1990s that evolved into the Agile movement.
Dependency Management Anxiety is Rooted in the Sprint Constraint
If you want to be Agile beyond the team level, beyond maturity level 1, then you have to remove the constraint of timeboxes to control batch size. A time constraint is a fantastic way to take a chaotic organization from the anarchy of maturity level 0 to the control and predictability of maturity level 1, but that is where it ends. Time-constrained sprints are an evolutionary cul-de-sac, a dead-end. You cannot scale Agile using timeboxed sprints!
The answer is to focus on quality and short delivery times, by using a WIP constraint, rather than a time constraint. Only the Kanban Method offers this for organizations struggling to scale Agile.
How to Start your Journey to Large-Scale Business Agility
I am now convinced that this pattern, taken from Kanban Maturity Model 2nd edition page 133, is a necessary step to large scale agility.
For each Scrum team, leave everything else alone, except you remove the sprint timebox – stories started dont have to finish within 2 weeks. Instead of limiting the time, limit the WIP, use a constant work-in-progress (CONWIP) limit. Initially, replenishment cadence and delivery cadence can be left synchronized with the former sprint cadence, eventually, you are likely to tune these to the needs of the business domain, the transaction, and coordination costs of holding the meetings, and the arrival rate of new information, that affects selection, sequencing, scheduling, and priority. This is your first step on a journey of evolutionary change, and your first step on the road to large scale business agility.
Relief from the Tyranny of the Timebox This change frees you from the tyranny of the timebox, and its three dysfunctions of upfront analysis, excessive and costly estimation, and heavy-weight dependency management.
Avoid: Will it fit within the timebox?
If things take, however long they take, then there is no need for upfront analysis to break an item into smaller items, no need to track complicated dependencies between sprints or across teams. Instead, let the work break down happen naturally once the work is committed for delivery. Track that work with a 2-tiered kanban board, parking lots, and avatars showing the involvement of shared services.
There is no need for awkward elaborate, big analysis upfront, estimation efforts. Simply track historical lead time through your workflow and use the lead time distribution chart to determine probabilistically how long something might take to complete.
Avoid: Will it get delayed by a dependency?
If things take, however long they take, then there is no need for upfront analysis to determine the need for specialists or expertise, nor do you need to constantly reorganize pursuing the Nirvana of the perfect cross-functional team that never needs assistance from outside. Let dependencies happen as you discover them, create visibility on to them, actively track them. Use a service-oriented approach to your organization design and utilize shared services for elite, specialist, market leading capabilities.
A General Solution for Dependency Management
If you need to pursue large-scale business agility, then you need to drop the use of time constraints to control batch size and improve quality and time-to-market. Instead you need to use WIP constraints and embrace the specialist shared services in your organization. Stop reorganizing! Start flowing work across a network of interdependent services!
To have a truly effective general solution for dependency management, you need to understand the opportunity cost of delay. You need to understand the urgency of a piece of work, and how dependency delays affect it. We have released that solution as an infographic poster in print-ready PDF format. You can also find it in Appendix F of the new Kanban Maturity Model 2nd edition available now.
If you are serious about enterprise scale business agility, we know how to take you there. Contact us for more information.
___________________________________
¹ Feature-driven Development, from Peter Coad and Jeff De Luca
² Iteration implies that something should be reworked. Iterative methods in software engineering such as Barry Boehm´s Spiral Method already existed in the literature long before Agile methodologies. Agile methods use incremental development with each piece of scope building on the last rather than reworking that which has gone before, although reworking to improve the fidelity of functionality, is both possible and feasible with Agile methods, it is unusual, particularly with Scrum.
³ Since, the original article the Scrum Guide has been modified to remove ¨commitment¨ and replace it with ¨forecast¨. While most organizations still use the term Sprint Commitment and customers intepret it as a promise, the official guide has modified it to what we might consider a ¨soft commitment¨, an indicator rather than a hard promise.
