We recently updated the Secure Funding goal diagram, shown below. The change was the addition of the Choose Funding Scope decision point, to recognize that the type of team you’re funding will affect how you fund said team. We have also published details about each decision point and its associated options, including the tradeoffs of each, at the Secure Funding overview page. This is information that we previously had only published in the first Disciplined Agile Delivery book but we’ve decided to start publishing updated versions of this information to the web to make it more accessible to you.
One of the things that a delivery team needs to do, often in collaboration with product management, is choose the release cadence of their product. This is an important aspect of, you guessed it, release planning. Your release cadence defines how often you release your solution both internally and externally into production (or the marketplace). The issue is how to determine how often the product should be released into production. In this blog we explore:
- Where are you deploying to?
- What affects release cadence?
- What release cadence choices do you have?
- What do we recommend?
Where Are You Deploying?
There are several target environments that you might choose to deploy to. These environments include:
- Demo environment(s). Many teams maintain a demo environment for their solution so that their stakeholders can see what has been developed to date at their leisure. Demo environments support transparency with your stakeholders, reduce the number of “one-off” requests by stakeholders for demos (because they can simply see the solution for themselves), and of course they provide a stable environment in which your teams can run demos.
- Testing environment(s). Many teams have their own testing environments, or they work with independent test teams with their own testing environments, or both. You should strive to test as often and early as possible, an implication being that want to deploy into your test environment(s) as often as you possibly can.
- Production/marketplace. Some teams will release their solutions into their production environments (or to someone else’s cloud) where end users will use the systems. In the case of commercial software companies they will release their solutions into the marketplace where they are then sold to customers. Throughout this blog whenever we use the term production we also mean the marketplace.
For the sake of terminology, deploying into demo or testing environments are often referred to as internal releases and into production as an external release.
What Affects Release Cadence?
There are several factors that affect the choice of release cadence:
- Stakeholder needs. How often do your stakeholders, and in particular your end users, want your solution to be released? This can be a difficult issue because very often your stakeholders might not be able to perceive what is appropriate for them. We’ve seen stakeholders ask for quarterly releases, be delighted when then get monthly releases, and then start asking for weekly releases once they realize the potential of modern agile strategies.
- Stakeholder capability to accept change. We like to think that more often is better, and in the vast majority of situations it is. As difficult to believe as this may seem, at the far extreme we’ve also seen some systems where the natural release cadence is once every three or four years because that’s the rate at which stakeholders are able to accept change. In this case the product was a transaction processing (TP) system infrastructure product, but we’ve heard similar stories about major database management systems (DBMSs) products too. Granted, a release cadence this long is very rare but it does happen in a small number of situations. Far more common is the mistaken belief by IT professionals that their stakeholders are unwilling or unable to accept shorter release cycles. We’ve seen numerous organizations where the IT people tell us that their stakeholders can’t handle anything more regular than a quarterly or bi-annual release, yet these same stakeholders regularly use commercial software that is updated several times a week.
- Your organizational culture. Some organizations, particularly those with an existing traditional release management team, often have release cultures that lean towards larger and less frequent releases. These organizations often have significant investments in legacy systems and insufficient investments in automated regression tests/checks. As a result releasing solutions into production tend to be seen as a risky endeavour. At the other extreme we’ve seen companies with more of a continuous delivery mindset that have a “release as swiftly and often as you can” culture. These organizations have typically invested heavily in code quality, automated regression testing, and automated deployment thus making deployment a simple and virtually risk-free effort.
- The team’s ability to deliver. Of course a primary determinant of your release cadence will be how often you’re able to actually produce a potentially consumable solution. This is affected by the skills of your team members, your ability to collaborate, your ability to vertically slice functionality into small features, and your delivery infrastructure.
- Your delivery infrastructure. How easy it is to release a potentially consumable solution into production is determined in part by your technical environment. This includes the extent of your automated regression tests, your automated deployment scripts, and your capability to monitor production. In general, the greater the level of automation the more often you can release.
- Your solution architecture. Is your solution architected to be released incrementally? Is it possible to enable/disable functionality at a granular level (perhaps via feature toggles or a similar technique)?
- The cost/risk to release. Cost and risk tend to go hand-in-hand when it comes to releasing solutions into production. This is because the more manual your release/deployment processes the more expensive they become and the more likely there are to be problems somewhere in the process. Conversely, the more you automate the overall deployment effort the cheaper it is to deploy and the less risky it becomes as you’re more likely to run into, and then automate a solution to, deployment problems. The less expensive and less risky it is to release your solution the more viable it becomes to release more often.
- Release cadence of other teams. Like it or not your team is likely dependent on the work of other teams. For example you may need web services being built by another team, and you can’t fully release your solution until those web services are available in production. We’ve written detailed articles about how to manage dependencies between agile/lean and traditional teams, dependencies between agile teams, and dependencies between agile and lean teams.
Release Cadence Choices
Table 1 lists many common release cadences, from more than annual to several times a day. It also lists the potential tradeoffs of each approach and indicates when you may want to adopt each one.
Table 1. Comparing external release cadence options.
|Strategy||Potential Advantages||Potential Disadvantages||When to Apply|
|Many times a day||Enables very short time to market
Enables teams to adapt quickly to changing stakeholder needs
Enables granular release of functionality
|Requires extensive continuous integration (CI) and continuous deployment (CD) automation
Requires high discipline to maintain quality
Your solution architecture must support toggling of features to enable deployment of larger functions as a collection of smaller features
|Effective for high-use systems, particularly those used by external customers in highly-competitive environments|
|Daily||Same as above
Provides a regular (daily) release cadence that is predictable
|Same as above||Same as above|
|Weekly||Provides a regular (weekly) release cadence that is predictable
Enables quick time to market and responsiveness to changing needs
|Same as above||Effective for high-use solutions, particularly e-commerce or BI/reporting systems
Appropriate for teams following the Lean lifecycle
|Monthly||Provides a regular (monthly) release cadence that is predictable
Enables quick time to market and responsiveness to changing needs
|Requires extensive continuous integration (CI) and continuous deployment (CD) automation
Requires high discipline to maintain quality
|Effective for medium-priority solutions
Appropriate for teams following the Agile/Basic lifecycle with one-week iterations or the Lean lifecycle
|Quarterly||Provides a regular (quarterly) release cadence that is predictable
Enables quick time to market and responsiveness to changing needs
Enables simpler requirements management practices (compared with longer release cadences) due to lower impact of a feature moving to the next release
This is a major milestone for teams moving towards an “advanced” lean-agile strategy as it motivates greater discipline.
|Requires continuous integration (CI)
Requires automated deployment strategies
|Effective for medium-priority solutions
Appropriate for teams following the Agile/Basic lifecycle with one or two week iterations
|Variable||Works well with a project mindset (although that’s questionable in and of itself)||Teams need to be able to judge when their work reaches the minimally marketable release (MMR) stage and the business value added exceeds cost of transition. This decision point is captured in the DAD project lifecycles by the “sufficient functionality” milestone
Politics can hamper this decision point. You should put an upper limit on the acceptable time between release
Stable teams assigned large “projects”
|Bi-annual||Good starting point for teams new to agile who are currently working on traditional projects with longer release cadences because it motivates adoption of disciplined strategies||Can be difficult for stakeholders who are used to less frequent releases||The team may need significant agile coaching as they will run into many of the “but we’re different and that agile stuff can’t possibly work here” type of problems|
|Annual||Provides a regular (annual) release cadence that is predictable
|Very risky, the team is likely to miss their date
Requires internal releases to obtain feedback
The deployment has likely become high risk because you do it so infrequently (self fulfilling problem)
|Appropriate for low priority systems or for high-risk deployments (note that the deployments may have become high-risk because you do them so infrequently)|
|More than annual||See annual
This is common for infrastructure systems, such as a database or transaction managers, that have many other systems highly dependent upon them
When it comes to releasing your solution, we have several recommendations for you to consider:
- Automate, automate, automate. The more you have automated, the lower the cost of deployment and the lower the risk. This enables you to release more often with confidence.
- Release internally very often. This is your opportunity to get good at releasing your solution, at squeezing out the cost and the risk.
- Release externally as often as possible. The faster and more often you can release into production the more competitive your organization will be.
- Always look for ways to release more often. Impressed with your ability to release once a month? Aim for bi-weekly. You’ve now releasing bi-weekly? What’s stopping you from releasing weekly? Weekly releases? Meh! Release daily! Your team is releasing daily grandpa? How about automatically releasing many times a day every time you have a working build?
- The Release Management Process blade describes the issues surrounding releases in a multi-team enterprise environment.
When a disciplined agile project or product team starts, one of the process goals which they will likely need to address is Identify Initial Technical Strategy. This is sometimes referred to as initial architecture envisioning or simply initial architecture modeling. This is an important process goal for several reasons. First, the team should think through, at least at a high level, their architecture so as to identify a viable strategy for moving forward into construction. A little bit of up-front thinking can increase your effectiveness as a team by getting you going in a good direction early in the lifecycle. Second, the team should strive to identify the existing organizational assets, such as web services, frameworks, or legacy data sources, that they can potentially leverage while producing the new solution desired by their stakeholders. By doing this you increase the chance of reuse, thereby avoiding adding technical debt into your organizational ecosystem, and more importantly you reduce the time and cost of delivering a new solution as the result of reuse. You will do this by working with your organization’s enterprise architects, if you have any. This is an aspect of Disciplined Agile’s philosophy of working in an enterprise aware manner.
The process goal diagram for Identify Initial Technical Strategy is shown below. The rounded rectangle indicates the goal, the squared rectangles indicate issues or process factors that you may need to consider, and the lists in the right hand column represent potential strategies or practices that you may choose to adopt to address those issues. The lists with an arrow to the left are ordered, indicating that in general the options at the top of the list are more preferable from an agile point of view than the options towards the bottom. The highlighted options (bolded and italicized) indicate default starting points for teams looking for a good place to start but who don’t want to invest a lot of time in process tailoring right now. Each of these practices/strategies has advantages and disadvantages, and none are perfect in all situations, which is why it is important to understand the options you have available to you.
Let’s consider each process factor:
- Choose the level of detail. How much effort should you put into capturing your architectural strategy, if any at all. A small co-located team may find that a high-level overview captured using whiteboard sketches is sufficient. A team that is geographically distributed across several locations will find that it needs to at least take digital snapshots of the sketches and share them (perhaps via a wiki) or even capture the diagrams using a drawing tool such as Visio or OmniGraffle. They may also find that they need to define the details of the interfaces to major components and services, a strategy often referred to as design by contract. A team in a life-critical regulatory environment may choose to capture more details, even to the point of doing big design up front (BDUF).
- Explore technology architecture. Technical aspects of your architecture may be captured via free form layered architecture diagrams, network diagrams, or UML deployment diagrams amongst others.
- Explore business architecture. Business or domain aspects may be explored via conceptual domain models, high-level process models, and UML component diagrams amongst others.
- Explore user interface (UI) architecture. For end users, the UI is the system. The implication is that you had better architect it well, and more important ensure that it is usable/consumable. You may explore the user interface architecture via a UI flow diagram/wireframe model, low-fidelity UI prototypes, or high-fidelity UI prototypes.
- Consider the future. A critical thing is for your architecture to be able to accommodate change in the future. The lean strategy is to consider these changes, and make architectural choices that best enable you to accommodate the changes when and if you need to, but DO NOT overbuild your solution now. In short, think but wait to act. To do this you need to consider the potential changes that could occur, often via “what if” discussions. If you need to capture these potential changes you might decide to write change cases.
- Apply a modeling strategy(ies). How will your team go about formulating their architecture? Will they hold informal modeling sessions in an agile modeling space or formal modeling sessions Joint Application Design (JAD) strategy? Or both?
- Select an architecture strategy. Will they identify a single technical strategy or several options which they will hope to explore and eventually choose from (or combine ideas from)?
- Identify a delivery strategy. Will the team be building a solution from scratch? Will they be evolving an existing solution? Will they be evolving a purchased, commercial off the shelf (COTS) package? Combinations thereof?
We want to share two important observations about this goal. First, this goal, along with Explore Initial Scope, Coordinate Activities, and Move Closer to a Deployable Release seem to take the brunt of your process tailoring efforts when working at scale. It really does seem to be one of those Pareto situations where 20% addresses 80% of the work, more on this in a future blog posting. As you saw in the discussion of the process issues, the process tailoring decisions that you make regarding this goal will vary greatly based on the various scaling factors. Second, as with all process goal diagrams, the one above doesn’t provide an exhaustive list of options although it does provide a pretty good start.
We’re firm believers that a team should tailor their strategy, including their team structure, their work environment, and their process, to reflect the situation that they find themselves in. When it comes to process tailoring, process goal diagrams not only help teams to identify the issues they need to consider they also summarize potential options available to them. Agile teams with a minimal bit of process guidance such as this are in a much better situation to tailor their approach than teams that are trying to figure it out on their own. The DA process decision framework provides this guidance.
When a disciplined agile project or product team starts one of the process goals which they will likely need to address is Explore Initial Scope. This is sometimes referred to as initially populating the backlog in the Scrum community, but as you’ll soon see there is far more to it than just doing that. This is an important goal for several reasons. First, your team needs to have at least a high level understanding of what they’re trying to achieve, they just don’t start coding. Second, in the vast majority of organizations IT delivery teams are asked fundamental questions such as what are you trying to achieve, how long will it take, and how much will it cost. Having an understanding of the scope of your effort is important input into answering those sorts of questions.
The process goal diagram for Explore Initial Scope is shown below. The rounded rectangle indicates the goal, the squared rectangles indicate issues or process factors that you may need to consider, and the lists in the right hand column represent potential strategies or practices that you may choose to adopt to address those issues. The lists with an arrow to the left are ordered, indicating that in general the options at the top of the list are more preferable from an agile point of view than the options towards the bottom. The highlighted options (bolded and italicized) indicate default starting points for teams looking for a good place to start but who don’t want to invest a lot of time in process tailoring right now. Each of these practices/strategies has advantages and disadvantages, and none are perfect in all situations, which is why it is important to understand the options you have available to you.
Let’s consider each process issue:
- Choose the level of detail. How much effort should you put into capturing the requirements, if any at all. A small co-located team may find that capturing user stories on index cards to be sufficient, whereas a team that is geographically distributed across several locations will find that it needs to capture point-form notes about each story using an electronic tool, and a team in a life-critical regulatory environment may need to capture even more detail to the point of doing big requirements up front (BRUF).
- Explore usage. Although much ado has been made of user stories, and they can be applied quite effectively in a range of situations, the fact is that they’re only one of several options for your team to explore usage of the solution (scenarios, personas, and use cases being other options).
- Explore the domain. Some teams will choose to do some domain modeling via a data model or class diagram, as well as address other views as appropriate.
- Explore the process. Many teams will discover that they need to explore the overall workflow, or business process, supported by their solution so as to help them better understand their usage requirements.
- Explore user interface (UI) needs. Many agile teams will also choose to create user interface prototypes, either low-fidelity UI prototypes using paper or even high-fidelity UI prototypes using a prototyping tool or code, particularly when they face a complex domain.
- Explore general requirements. There are several types of functional requirements modeling techniques that can be valuable that don’t fit well into the previous categories.
- Explore non-functional requirements. How will non-functional requirements pertaining to availablity, security, performance, and many other issues be addressed? Teams in straightforward situations may find that capturing them as technical stories may be sufficient. Teams facing technical complexity, and sometimes even domain complexity, soon discover that they need a more sophisticated strategy.
- Apply modeling strategy(ies). How will your team go about working with stakeholders to elicit/discover their perceived needs? Will they hold informal modeling sessions in an agile modeling space? Will they hold formal modeling sessions, perhaps following a Joint Application Design (JAD) strategy? Will they interview people one-on-one? Combinations thereof?
- Choose a work item management strategy. Early in the project you will want to determine how you intend to address changing stakeholder needs throughout the project as this will affect how you address the other process issues in this list. For example, do you intend to adopt Scrum’s value-driven product backlog strategy, DAD’s risk-value driven work item list, a lean work item pool strategy (as followed by DAD’s lean lifecycle), or even a formal approach? A team in a strict regulatory environment may be required to have a more formal approach to change management than a team without this restriction.
I wanted to share two important observations about this goal. First, this goal, along with Identify Initial Technical Strategy, Coordinate Activities, and Move Closer to a Deployable Release seem to take the brunt of your process tailoring efforts when working at scale. It really does seem to be one of those Pareto situations where 20% addresses 80% of the work, more on this in a future blog posting. As you saw in the discussion of the process issues, the process tailoring decisions that you make regarding this goal will vary greatly based on the various scaling factors. Second, as with all process goal diagrams, the one above doesn’t provide an exhaustive list of options although it does provide a pretty good start.
I’m a firm believer that a team should tailor their strategy, including their team structure, their work environment, and their process, to reflect the situation that they find themselves in. When it comes to process tailoring, process goal diagrams not only help teams to identify the issues they need to consider they also summarize potential options available to them. Agile teams with a minimal bit of process guidance such as this are in a much better situation to tailor their approach that teams that are trying to figure it out on their own. The DA process decision framework provides this guidance.
This post is a follow-up to Comparing DAD to the Rational Unified Process (RUP) – Part 1. In that post I described in some detail why Disciplined Agile Delivery (DAD) is not “Agile RUP”. DAD is quite different in both approach and content. There are however some very good principles that the Unified Process (UP) incorporates that are not part of mainstream agile methods. This post describes what parts of the UP made it into the DAD process decision framework.
DAD suggests a full delivery lifecycle approach similar to RUP. DAD recognizes that despite some agile rhetoric projects do indeed go through specific phases. RUP explicitly has four phases for Inception, Elaboration, Construction, and Transition. For reasons that I described in the last post, DAD does not include an explicit Elaboration phase. However the milestone for Elaboration is still in DAD which I will describe shortly. As the DAD basic lifecycle diagram shows, DAD has three of the four RUP phases.
- The Inception phase. An important aspect of DAD is its explicit inclusion of an Inception phase where project initiation activities occur. As Scott Ambler says in one of his posts “Although phase tends to be a swear word within the agile community, the reality is that the vast majority of teams do some up front work at the beginning of a project. While some people will mistakenly refer to this effort as Sprint/Iteration 0 it is easy to observe that on average this effort takes longer than the general perception (the 2009 Agile Project Initiation survey found the average agile team spends 3.9 weeks in Inception)”. So in DAD’s Inception phase (usually one iteration) we do some very lightweight visioning activities to properly frame the project. The milestone for this phase is to obtain “Stakeholder consensus” on how to proceed. In the book we describe various strategies to get through the Inception phase as quickly as possible, what needs to be done, and how to get stakeholders consensus.
- The Construction phase. This phase can be viewed as a set of iterations (Sprints in Scrum parlance) to build increments of the solution. Within each iteration the team applies a hybrid of practices from Scrum, XP, Agile modeling, Agile data, and other methods to deliver the solution. DAD recommends a risk-value approach of prioritizing work in the early iterations which draws from the RUP principle of mitigating risk as early as possible in the project by proving the architecture with a working solution. We therefore balance delivering high-value work with delivering work related to mitigating these architectural risks. Ideally we deliver stories/features in the early iteration that deliver functionality related to both high business value and risk mitigation (hence DAD’s “risk-value” lifecycle). It is worthwhile to have a checkpoint at the end of the early iterations to verify that indeed our technical risks have been addressed. DAD has an explicit milestone for this called “Proven architecture”. This is similar to the RUP Elaboration milestone without risking the confusion that the Elaboration phase often caused for RUP implementations. All agile methods seek to deliver value into the hands of the stakeholders as quickly as possible. In many if not most large enterprises it is difficult to actually deliver new increments of the solution at the end of each iteration. DAD therefore recognizes this reality and assumes that in most cases there will be a number of iterations of Construction before the solution is actually deployed to the customer. As we make clear in the book, although this is the classic DAD pattern, you should strive to be able to release your solution on a much more frequent basis in the spirit of achieving the goal of “continuous delivery”. The milestone for the end of Construction is that we have “Sufficient functionality” to deploy to the stakeholders. This is the same milestone as the RUP’s Construction milestone. During the Construction phase it may make sense to periodically review the progress of the project against the vision agreed to in Inception and potentially adjust course. These optional milestones in DAD are referred to as “Project viability”.
- The Transition phase. DAD recognizes that for sophisticated enterprise agile projects often deploying the solution to the stakeholders is not a trivial exercise. To account for this reality DAD incorporates the RUP Transition phase which is usually one short iteration. As DAD teams, as well as the enterprise overall streamline their deployment processes this phase should become shorter and ideally disappear over time as continuous deployment becomes possible. RUP’s Transition milestone is achieved when the customer is satisfied and self-sufficient. DAD changes this to “Delighted stakeholders”. This is similar to lean’s delighted customers but we recognize that in an enterprise there are more stakeholders to delight than just customers, such as production support for instance. One aspect of RUP’s Transition phase is that it is not clear on when during the phase deployments actually take place. Clearly stakeholders aren’t delighted and satisfied the day the solution goes “live”. There is usually a period of stabilization, tuning, training etc. before the stakeholders are completely happy. So DAD has a mid-Transition milestone called “Production ready”. Some people formalize this as a “go/no go” decision.
So in summary, DAD frames an agile project within the context of an end-to-end risk-value lifecycle with specific milestones to ensure that the project is progressing appropriately. These checkpoints give specific opportunities to change course, adapt, and progress into the next phases of the project. While the lifecycle is similar to that of RUP, as described in Part 1 of this post it is important to realize that the actual work performed within the iterations is quite different and far more agile than a typical RUP project.
At Scott Ambler + Associates we are getting a lot of inquiries from companies seeking help to move from RUP to the more agile yet disciplined approach that DAD provides.
An important aspect Disciplined Agile Delivery (DAD) is its explicit inclusion of an Inception phase where project initiation activities occur. Although phase tends to be a swear word within the agile community, the reality is that the vast majority of teams do some up front work at the beginning of a project. Some people will mistakenly refer to this effort this Sprint/Iteration 0 it is easy to observe that on average this effort takes longer than a single iteration (the 2009 Agile Project Initiation survey found the average agile team spends 3.9 weeks in Inception and the November 2010 Agile State of the Art survey found that agile teams have Construction iterations of a bit more than 2 weeks in length).
Regardless of terminology, agile teams are doing some up front work. Part of that initial work is identifying an initial technical architecture, typically via some initial architecture envisioning http://www.agilemodeling.com/essays/initialArchitectureModeling.htm. Because your architecture should be based on actual requirements, otherwise you’re “hacking in the large”, your team will also be doing some initial requirements envisioning http://www.agilemodeling.com/essays/initialRequirementsModeling.htm in parallel. Your architecture will be driven in part by functional requirements but more often the non-functional requirements, also called quality of service (QoS) or simply quality requirements. Some potential quality requirements are depicted in the figure below (this figure is taken from the Disciplined Agile Delivery book but was first published in Agile Architecture Strategies ).
Some architects mistakenly believe that you need to do detailed up front modeling to capture these quality requirements and then act upon them. This not only isn’t true it also proves to be quite risky in practice, see my discussion about Big Modeling Up Front (BMUF) for more details. Disciplined agilists instead will do just enough initial modeling up front and then address the details on a just-in-time (JIT) basis throughout construction. Of course it’s important to recognize that just enough will vary depending on the context of the situation, teams finding themselves at scale will need to do a bit more modeling than those who don’t. It’s also important to recognize that to address non-functional requirements throughout construction that you need to have more than just architectural modeling skills. This topic will be the focus of my next blog posting in this series.
Recently at the Scott W. Ambler + Associates site we received a series of questions from someone who wanted to better understand how architecture issues are addressed on agile project teams. It seemed to me that the questions were sufficiently generic to warrant a public response instead of a private one. So, over the next few days I’m going to write several blog postings here to address the issues that were brought up in the questions. It’s important to note that I will be answering from the point of view of Disciplined Agile Delivery (DAD), and not agile in general. Other agile methods may provide different advice than DAD does on this subject, or no advice at all in some cases.
The goal of the first blog posting in this series is to address several potential misconceptions that appeared in the email. I want to start here so as to lay a sensible foundation for the follow-on postings.
Partial Misconception #1: Agile can be prefixed in iteration 0 by architectural design
I’ve named this a “partial misconception” for a few reasons:
- Disciplined agile teams do some up-front work. This is called the Inception Phase in DAD, although other methods may refer to it as iteration/sprint 0, warm up, initiation, or other names. Up-front work is an explicit part of DAD.
- Iteration 0 isn’t an accurate term. Although I have used this term in the past when discussing project initiation, the reality is that the average agile team spends about a month doing project initiation activities whereas the average iteration length is two weeks. So, Inception really isn’t a proper iteration.
- Inception is more than just architecture. Several activities typically occur at this point in time, particularly initial architecture envisioning, initial requirements envisioning, initial release planning, and putting the team together to name a few things.
Partial Misconception #2: On principle, Agile is against “big” anything
This is also a “partial misconception” for several reasons:
- There is in fact a lot of agile rhetoric against big artifacts. It’s very easy to find agile writings about the challenges with big requirements up front (BRUF), big modeling up front (BMUF) in general, and detailed up front planning for instance.
- Disciplined agile is against needless waste, not “big” things. Many traditional modeling and planning practices prove to be quite wasteful in practice. A serious cultural challenge that the traditional community has is that they are afraid to throw out the bathwater because they assume that the baby will go with it. I believe that Disciplined Agile Delivery (DAD), and Agile Modeling before it, make it quite clear that it’s possible to gain the benefit of thinking before doing without taking on the very serious problems around doing too much thinking before doing. So, have the discipline to keep the thinking “baby” yet discard any needless documentation “bathwater”.
- In rare situations it’s appropriate to create “big” artifacts. Disciplined agilists aim for sufficient artifacts, the size of which will depend on the context of the situation that your team finds itself in. In a recent article for Dr. Dobb’s Journal, Disciplined Agile Architecture, I explicitly explored how initial architecture envisioning on an agile project may result in “big” artifacts in some situations. These situations are very rare mind you, ignoring cultural imperatives to create big artifacts because some people still haven’t made the jump to a disciplined agile approach, but they do happen. One of the strengths of the DAD process decision framework is that it is goal driven, not prescriptive, and explicitly explores the tradeoffs surrounding the amount of detail to capture and when to do so.
Partial Misconception #3: Refactoring system architecture beyond mid-implementation is much more expensive than refactoring components
Once again, this is a partial misconception. I suspect part of the problem is a lack of understanding of what refactoring is really all about, a recurring problem with experienced traditionalists, and part because of a lack of understanding of how architecture is address by disciplined agile teams. Some thoughts:
- Refactorings are simple, not difficult. The goal of refactoring is to make SMALL changes to your design that improve the quality without changing the semantics of the design in a practical manner. This is true of code refactorings, database refactorings, user interface refactorings, and other types of refactorings. Small changes are inexpensive to make given the appropriate skills, tools, and organizational environment.
- Architectural rework (not refactoring) is often difficult. Rework, or rewrites, are very large changes the goal of which is typically to replace large portions of your solution. Yes, the later in the lifecycle such rework occurs very likely the more expensive it will be because you’ve built more based on that architecture that is now being reworked. This is a general issue, not just an agile one.
- Disciplined agile teams get going in the right direction to begin with. The practice of initial architecture envisioning, which we describe in detail in Chapter 9 of Disciplined Agile Delivery, aims to think through the architectural strategy before getting into construction.
- Disciplined agile teams prove their architecture works early. The first construction milestone, prove the architecture, reduces the risk of architectural rework. The goal is to prove that the architecture works by building a working end-to-end skeleton of the solution which implements critical/difficult technical requirements. This is an agile “fail fast” strategy, or as we say in DAD a “succeed early” strategy, that reduces technical risk on your project. As an aside, including explicit light-weight milestones such as this is one of many agile governance aspects built right into DAD.
- Disciplined agile teams have an architectural role. This role is called Architecture Owner and one of the responsibilities of the person in this role is to guide the team in architectural issues throughout the entire DAD lifecycle.
- There are no guarantees. No matter how smart your approach, there’s still a chance that rework can happen. For instance, you can be mid-way through a project and the vendor of a major architectural component of your solution decides to withdraw it from the market. Or the vendor goes out of business. Or perhaps your firm is taken over by another firm and the new owners decide to inflict, oops I mean bless you with, their architectural strategy. Stuff happens. Once again, this is a general issue, not specifically an agile one.
- Quality decreases the cost of rework. Disciplined agilists will write high-quality code, with a full regression test suite in place, at all times during Construction. It’s easier to rework high quality artifacts compared with low quality artifacts, so if you get stuck having to perform rework at least the pain is minimized. My article Agile Testing and Quality Strategies overviews many techniques.
In short, disciplined agile teams do what they can to avoid architectural rework to begin with by having an explicit architecture owner role who focuses on architectural issues throughout the entire lifecycle, by identifying a viable architectural strategy early in the project, proving that architectural strategy works early in Construction, and producing high-quality artifacts throughout the lifecycle that are easier to rework if needed. With continuous documentation practices and a focus on producing artifacts which are just sufficient enough for the situation at hand, this proves to be far more effective than traditional strategies that assume you require large up-front investments in “big” artifacts, that rely on validation techniques such as architecture reviews instead of the far more concrete feedback of working code, and that often leave quality strategies to the end of the lifecycle (thereby increasing the cost of any rework).
I plan two follow-on blog postings in this series, one exploring how initial architecture envisioning works and one about how to address initial quality requirements (also called non-functional requirements or quality of service requirements) on disciplined agile projects. Stay tuned!
At Scott W. Ambler + Associates we offer a one-day workshop entitled Agile Architecture: A Disciplined Approach that you should consider if you’re interested in this topic. We also offer coaching and mentoring services around agile architecture.
The Disciplined Agile Delivery (DAD) process framework includes an explicit Inception phase – sometimes called a project initiation phase, startup phase, or iteration/sprint zero – which is conducted before actually starting to build a solution. The goals of this phase include: clarifying the business problem that needs to be solved; identifying a viable technical solution; planning your approach; setting up your work environment and team; and gaining stakeholder concurrence that it makes sense to proceed with investing in the implementation of the chosen strategy. These goals are listed in the following diagram.
In the Disciplined Agile Delivery book we devoted a lot space to describing how to effectively initiate a DAD project. Unfortunately in our experience we have seen many organizations that are still new to agile treat this phase as an opportunity to do massive amounts of upfront documentation in the form of project plans, charters, and requirements specifications. Some people have referred to the practice of doing too much temporary documentation up front on an agile project as Water-Scrum-Fall. We cannot stress enough that this is NOT the intent of the Inception phase. While we provide many alternatives for documenting your vision in Inception, from very heavy to very light, you should take a minimalist approach to this phase and strive to reach the stakeholder consensus milestone as quickly as possible.
According to the 2013 Agile Project Initiation survey the average agile team invests about 4 weeks performing project initiation activities, including initial requirements envisioning, initial architecture envisioning, forming the team, initial release planning, and so on. Of course this is just an average, some respondents reported investing less than a week to do so and some reporting investing more than two months – the amount of time required varies depending on the complexity of the effort, your stakeholders’ understanding of their requirements, your team’s understanding of the solution architecture, whether this is a new solution or merely a new release of an existing solution, and many others.
If you are spending more than a few weeks on this phase, you may be regressing to a Water-Scrum-Fall approach. It takes discipline to be aware of this trap and to streamline your approach as much as possible. You can do this in several ways:
- Recognize that the goal is to get going in the right direction. For any project or product of reasonable complexity you want to spend a bit of time up front ensuring that you understand the problem and have what you believe to be a viable strategy to addressing that problem. This doesn’t imply that you need a comprehensive requirements specification, a detailed project plan, nor a comprehensive architecture model but it does mean you need to do a bit of initial thinking and organizing. In a small handful of cases, typically at scale, you may find that your team does require detailed models and plans but this is a very uncommon exception (regardless of what traditional THEORY may claim).
- Educate people that details can safely come later. If you have the ability to plan or model something in detail today, won’t you also have that same ability a few months from now when you actually need those details? Of course you do. In lean software development they recommend deferring decisions – including planning decisions, detailed design decisions, and even requirements – until the most appropriate moment. The observation is that by waiting you can make a better decision because you have better information at hand. This strategy of course assumes that you’re able to overcome basic logistical problems such as having the appropriate people available at the time to help explore an issue, provide requisite information, and make the decision. It’s far less risky, and far less expensive, in most cases to address basic logistical issues than it is to apply the process band-aid of writing detailed documentation at the beginning of a project.
- Promote a sense of urgency. This is the most important thing that you can do. Just as there is risk associated with not sufficiently thinking about your strategy for approaching a new project or product there are also risks associated with doing too much up front work. My experience is that far too many IT professionals are complacent regarding the risks associated with the project initiation activities of the Inception phase. The longer you put off building a consumable solution the greater the risk that you’re building the wrong thing.
- Keep your modeling efforts as light as possible. You very likely need to do some initial requirements envisioning and architecture envisioning early in the project lifecycle to help you think through what you’re doing. But in most cases this modeling should be high level and light, not detailed and heavy. In every project I’ve ever been involved with the team has been asked to identify what they’re going to deliver (at least giving a rough sense of the scope) and how they’re going to do it (at least providing a rough idea of the technical strategy) to secure funding for construction. In short, they need to do a bit of up front thinking. You will often find that you need to reign-in some of your staff who are experienced with traditional approaches to modeling and specification. These people have a lot of value to add to your project, modeling is an important skill needed on disciplined agile teams, but they may need help keeping their approach light-weight and incremental. The details can come later. See the process goals Identify Initial Scope and Identify Initial Technical Strategy for more thoughts on this subject.
- Keep your planning efforts as light as possible. Similarly you need to invest some time in high-level release planning to answer basic questions such as how long do you believe (roughly) it will take to get a release of your solution deployed and how much (roughly) this will cost. Planning details can come throughout the Construction phase when it’s more appropriate to invest in such decisions.
I think that it’s very clear that the secret to keeping Inception short is to have the discipline to know that you need to invest some time thinking your approach through but that you want to avoid getting bogged down in too many details. You need the discipline to do some planning but not too much. You need the discipline to do some modeling but not too much. You need the discipline to get going in the right direction knowing that the details will come out in time.
A few days ago I wrote about ranged burndown charts. Interestingly, if you track the ranges over time you end up with a chart such as the one below which corresponds to the estimating cone of uncertainty (depicted by the dashed lines). It’s interesting to note that this example includes two common occurrences that you’ll see. First, during iterations one and two the gross and net velocities were the same because no new functionality had been identified yet, resulting in an unranged estimate. Second, iteration eight had a very small net velocity because the amount of new functionality was almost as much as the amount implemented, giving a huge estimation range due to the small net velocity.