Category Archives: Transition iteration

Comparing DAD to the Rational Unified Process (RUP) – Part 2

This post is a follow-up to Comparing DAD to the Rational Unified Process (RUP) – Part 1.  In that post I described in some detail why Disciplined Agile Delivery (DAD) is not “Agile RUP”.  DAD is quite different in both approach and content.  There are however some very good principles that the Unified Process (UP) incorporates that are not part of mainstream agile methods.  This post describes what parts of the UP made it into the DAD process decision framework.

DAD suggests a full delivery lifecycle approach similar to RUP.  DAD recognizes that despite some agile rhetoric projects do indeed go through specific phases.  RUP explicitly has four phases for Inception, Elaboration, Construction, and Transition.  For reasons that I described in the last post, DAD does not include an explicit Elaboration phase.  However the milestone for Elaboration is still in DAD which I will describe shortly.  As the DAD basic lifecycle diagram shows, DAD has three of the four RUP phases.

Disciplined Agile Lifecycle Basic

  • The Inception phase.  An important aspect  of DAD is its explicit inclusion of an Inception phase where project initiation activities occur.  As Scott Ambler says in one of his posts “Although phase tends to be a swear word within the agile community, the reality is that the vast majority of teams do some up front work at the beginning of a project.  While some people will mistakenly refer to this effort as Sprint/Iteration 0 it is easy to observe that on average this effort takes longer than the general perception (the 2009 Agile Project Initiation survey  found the average agile team spends 3.9 weeks in Inception)”.  So in DAD’s Inception phase (usually one iteration) we do some very lightweight visioning activities to properly frame the project.  The milestone for this phase is to obtain “Stakeholder consensus” on how to proceed.  In the book we describe various strategies to get through the Inception phase as quickly as possible, what needs to be done, and how to get stakeholders consensus.
  • The Construction phase.  This phase can be viewed as a set of iterations (Sprints in Scrum parlance) to build increments of the solution.  Within each iteration the team applies a hybrid of practices from Scrum, XP, Agile modeling, Agile data, and other methods to deliver the solution.  DAD recommends a risk-value approach of prioritizing work in the early iterations which draws from the RUP principle of mitigating risk as early as possible in the project by proving the architecture with a working solution.  We therefore balance delivering high-value work with delivering work related to mitigating these architectural risks.  Ideally we deliver stories/features in the early iteration that deliver functionality related to both high business value and risk mitigation (hence DAD’s “risk-value” lifecycle). It is worthwhile to have a checkpoint at the end of the early iterations to verify that indeed our technical risks have been addressed.  DAD has an explicit milestone for this called “Proven architecture”.  This is similar to the RUP Elaboration milestone without risking the confusion that the Elaboration phase often caused for RUP implementations.  All agile methods seek to deliver value into the hands of the stakeholders as quickly as possible.  In many if not most large enterprises it is difficult to actually deliver new increments of the solution at the end of each iteration.  DAD therefore recognizes this reality and assumes that in most cases there will be a number of iterations of Construction before the solution is actually deployed to the customer.  As we make clear in the book, although this is the classic DAD pattern, you should strive to be able to release your solution on a much more frequent basis in the spirit of  achieving the goal of “continuous delivery”.  The milestone for the end of Construction is that we have “Sufficient functionality” to deploy to the stakeholders.  This is the same milestone as the RUP’s Construction milestone.  During the Construction phase it may make sense to periodically review the progress of the project against the vision agreed to in Inception and potentially adjust course.  These optional milestones in DAD are referred to as “Project viability”.
  • The Transition phase.  DAD recognizes that for sophisticated enterprise agile projects often deploying the solution to the stakeholders is not a trivial exercise.  To account for this reality DAD incorporates the RUP Transition phase which is usually one short iteration.  As DAD teams, as well as the enterprise overall streamline their deployment processes this phase should become shorter and ideally disappear over time as continuous deployment becomes possible.  RUP’s Transition milestone is achieved when the customer is satisfied and self-sufficient.  DAD changes this to “Delighted stakeholders”.  This is similar to lean’s delighted customers but we recognize that in an enterprise there are more stakeholders to delight than just customers, such as production support for instance.  One aspect of RUP’s Transition phase is that it is not clear on when during the phase deployments actually take place.  Clearly stakeholders aren’t delighted and satisfied the day the solution goes “live”.  There is usually a period of stabilization, tuning, training etc. before the stakeholders are completely happy.  So DAD has a mid-Transition milestone called “Production ready”.  Some people formalize this as a “go/no go” decision.

So in summary, DAD frames an agile project within the context of an end-to-end risk-value lifecycle with specific milestones to ensure that the project is progressing appropriately.  These checkpoints give specific opportunities to change course, adapt, and progress into the next phases of the project.  While the lifecycle is similar to that of RUP, as described in Part 1 of this post it is important to realize that the actual work performed within the iterations is quite different and far more agile than a typical RUP project.

At Scott Ambler + Associates we are getting a lot of inquiries from companies seeking help to move from RUP to the more agile yet disciplined approach that DAD provides.

It Requires Discipline to Evolve Transition From a Phase to an Activity

Because Disciplined Agile Delivery (DAD) addresses the full delivery lifecycle it explicitly addresses the effort required to transition your solution into production, or in the case of product teams into the marketplace.  This transition effort may be referred to as release, deployment, or even the “end game”.  For teams relatively new to agile this transition effort is a phase, or if you don’t like the term phase then an iteration which very likely has a different time frame than construction iterations.  However, as we indicate in the DAD book, as teams gain more experience with agile and lean techniques the “transition phase” can be evolved into a “transition activity” with a little bit of discipline.  That is the focus of this blog posting.

The DAD framework is goal-driven, not prescriptive, and as a result it describes the transition effort in terms of goals:

  1. Ensure that the solution is ready to deploy.  To achieve this goal you will need to address several issues, such as planning the transition effort, verifying that the solution meets the stakeholder needs for the current release, and validating that the solution is of sufficient quality.  As with other DAD goals, there are several ways each of these issues can be addressed, each with advantages and disadvantages.  There is no single strategy that’s right for all situations.
  2. Ensure that stakeholders are ready to receive the solution.  Similarly, there are several issues you need to consider when addressing this goal, including how you communicate that the release is coming, how to train and educate stakeholders in the new solution, and how to prepare the operations and support environment for the new release (one of the many aspects of a disciplined DevOps strategy built right into DAD).
  3. Deploy the solution.  Issues to consider when achieving this goal is identifying what should be released, how you can back out the release if you run into trouble, and how to determine that the release was successful.  The last issue, determining success, isn’t as obvious as you would think.  Does success mean that the solution is up and running in production?  Or does success mean that you have satisfied, or better yet, delighted stakeholders?

It is straightforward to empirically observe that the complexity of your transition effort can vary depending on the context of your situation.  For example, a simple standalone application such as a website can be easily deployed regularly because it effectively involves copying some files to a server (farm).  Organizations in this sort of situation may choose to deploy their software many times a day.  However, as we showed in the DAD book, for non-trivial enterprise deployments the transition effort can be significant.    In fact, the November 2010 Agile State of the Art survey found that the average agile team spent six weeks in their transition efforts, with some respondents indicating spending one day or less and some indicating twelve weeks or more.  There are several reasons why this happens:

  1. Transition isn’t agile yet. Many organizations are adopting agile techniques for construction but not for other aspects of the delivery lifecycle.  This is often referred to as “Water-Scrum-Fall”.  As a result these organizations are inflicting traditional, labor intensive transition practices on their agile teams.
  2. We’re dealing with more than just software.  It’s often a bit more difficult than simply copying some files onto a production server.  Mainstream agile methods describe the goal of creating “shippable software” at the end of every iteration.   That’s a wonderful, overly simplistic philosophy that doesn’t hit the mark in practice. In DAD we prefer to use the phrase “consumable solution” indicating that shippable does not necessarily mean usable software.  Furthermore, in many situations more needs to be delivered than just software – there may also be hardware upgrades, the business processes surrounding the use of the system may evolve, and even the organization structure of the people using the system may evolve.  In short, disciplined agilists adopt a solutions over software mindset.
  3. Transition occurs infrequently.  Some agile project teams adopt a long release cycle, often six months or more, and as a result don’t have sufficient opportunities or motivation to get good at deployment.  Sometimes this is due to the nature of the solution they’re working on but more often than not it’s because of the cost of transition – the greater the effort to transition into production the greater the cost to do so, therefore you need to spend more time building the solution to add sufficient value to justify the cost of transition.
  4. Lack of vision.  Many IT organizations don’t believe it’s possible that releasing solutions into production can be evolve from a multi-week phase to a multi-hour, or even multi-minute, activity.  Or they realize it’s possible for others to do so but not them (I find it amazing how many people believe they’re in a special situation or blame others, often senior management, for not being able to adopt fundamental strategies that would improve their approach).

Because of the potential complexities of releasing a solution in most mid-to-large sized organizations the deployment of solutions is carefully controlled.  This is particularly true when the solutions share architectures and have project interdependencies, one of the reasons why DAD promotes the need for enterprise awareness within agile teams.  For many reasons release cycles to your stakeholders are less frequent that you would like because of existing complexities within the environment.  However, the ability to frequently deploy value to your stakeholders is a competitive advantage; therefore you should reduce the release cycle as much as possible.  To do so requires a great degree of discipline in areas such as:

  1. Quality practices.  First and foremost, testing and any related fixing should be done as much as possible during construction iterations.  Transition is not a “stabilization phase” nor is it a “test phase”, it’s a deployment effort.  Granted, you should still run your regression test suite(s) one last time as part of your overall deployment effort, with the potential that if tests fail you will need to do some fixing, but this should be a minimal effort.  The implication is that quality practices – including testing, refactoring, reviews, code analysis, and so on – be performed continually throughout construction.
  2. Pre-production integration and deployment testing.   These are two of the most challenging types of testing, and sadly they’re often given short-shrift in the agile literature.   The goal of pre-production integration testing, sometimes called production testing, is to test the solution in the best approximation of the production environment as possible.  This can be particularly challenging in larger organizations where hundreds of systems are already running in production and dozens are being worked on at any given time.  To make this testing economical during construction many organizations find that they need to go beyond the “whole team testing” approach with a parallel independent testing team that focuses on complex forms of testing that development teams can struggle with.  Similarly deployment testing can also be a challenge because its goal is to determine whether you can successfully deploy into production.
  3. Regular coordination between project teams and with operations and support staff. Throughout construction you will want to ensure that the development, operations, and support teams are all aligned.   One aspect of DAD’s enterprise awareness is to include DevOps strategies, such as treating operations and support staff as key stakeholders that you need to work closely with throughout the lifecycle, in the process itself.  The better the coordination the smoother, and therefore quicker, transition will go.
  4. Adoption of continuous deployment practices.  The fundamental idea behind continuous deployment is that the more often you choose to deploy the better at it you will get.  This happens because you have more opportunities to hone your deployment skills and also because you’re motivated to find ways to simplify or automate deployment activities.

I’ve seen teams evolve transition phases of several weeks down to several hours through adoption of the disciplined strategies mentioned above.   A disciplined agile team may start with relatively long Inception, Construction, and Transition phases and over time shrink all three down.   Over time the Inception phase mostly disappears, particularly if you maintain team consistency between releases, and as I’ve argued in this posting the Transition phase shrinks to a very small activity.  When deployment becomes inexpensive it enables you to have shorter construction phases and thus more regular releases – teams go from annual releases to bi-annual, then to quarterly releases, then monthly, weekly, and yes, sometimes even daily.  Your team will need to choose a release cadence that makes sense for you.

These days it is fairly easy to observe multi-billion dollar companies, particularly e-commerce companies but even staid organizations such as financial institutions, deploy on a monthly, weekly, and even daily basis.  If other organizations choose to work this way then why can’t you?

Injecting Transition iterations into your Release plan

For complex agile projects, deploying (or transitioning) the release to a “live” environment for the users is seldom a trivial exercise.  Most large enterprises have defined milestones, gates, and or reviews that need to be coordinated with many diverse stakeholder groups such as users, governance bodies (such as Project Management Offices), DevOps, and marketing before the application can be released.  In DAD we therefore describe a distinct set of activities to prepare our stakeholders for the release and support of the new application.  This could include activities such as user training, data conversion, documentation, hardware deployment, preparing customer support, database tuning, and last minute defect fixing.  To recognize the clear difference from a typical Construction iteration, we describe iterations dedicated to deployment as “Transition” iterations.  The illustration shows how we can inject iterations into our release schedule to deploy increments to the user community.  In this example, after the fourth Construction iteration, we may have an additional set of features representing a minimally marketable release that is worthy of a Transition iteration to deliver the value to the users.

When we have an application that needs to be delivered over multiple releases, following the Transition phase we may start work on a new release by continuing the Construction phase.  Since we would typically have an existing work item list (backlog) of outstanding requirements, we can simply pull more requirements off this stack and continue to build more functionality.  In this way, it may not be necessary to have another Inception phase unless the vision has materially changed and needs to be revisited.  Your organization may, however choose to run small Inception iterations at the beginning of each new release.

In this example, after the fourth Construction iteration, we may have an additional set of features representing a minimally marketable release that is worthy of a Transition iteration to deliver the value to the users.

Some people mistakenly interpret DAD as one deployment to the customer at the end of the project.  If possible, we prefer to deploy frequently, in support of the agile alliance principle “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”.  Our minor quibble with this principle is that what we deliver to the customer is a “solution”, which may not only include software, but also business process changes, training, or marketing activities, for example.  Our experience is that large projects typically involve a lot of work beyond the software itself which can also benefit from agile collaborative practices.

Injecting Transition iterations to deploy increments to customers

While the illustration refers to each iteration delivering a “potentially shippable” increment, which is what the agile community typically calls it, we actually prefer to use the term “consumable” to indicate that it not only works, but it is also actually usable by the customer.
________________

In our upcoming book we discuss patterns like these with considerations for what might make sense for your program or project in various circumstances.

Product owner issues in the Transition phase of large agile projects

Captain Barbossa

Mainstream agile methods would suggest that we should have one product owner, being the “one” voice of the customer for matters related to release/iteration scope, priorities, and requirements clarification.

The reality that I see on most enterprise agile projects is that the product owner cannot possibly manage the work item list themselves.  In fact, they are often not the best person to do this prioritization.  I wish it were as easy as managing epics and user stories.

On my current project, we are in the process of taking ownership of a very large application built offshore.  In DAD, we would describe this as the Transition phase of the product as we deploy the solution to our stakeholders.  We will take over the build, deployment, support and enhancement going forward.  This is a team of 20+ with responsibilities spanning development, infrastructure, and devops.  Due to the complexity of the rollout we are doing it over a number of Transition iterations.

A simplistic view would suggest that the Program Manager is our product owner.  At the end of the day he may have ultimate responsibility, but he is not collocated, and deals with the larger issues of the project, such as vendor management, governance, funding, resourcing, and other related issues.  He delegates responsibility for the details to domain leads for the business, development, infrastructure and support.

My approach as the overall agile team lead is to treat the these traditional leads as product owners.  They each have their own work items and priorities such as:

Development team:

  • setting up VM images for development support of code
  • determining a source code management strategy for parallel streams of maintenance and enhancements
  • automation of continuous integration & deployment

Support team:

  • establishing a customer support strategy, with triage process, roles, on call schedules etc.

Infrastructure/DevOps team:

  • setting up automated jobs
  • performance tuning

Business:

  • change requests/stories
  • defects

All:

  • knowledge transfer between vendor and support team
  • satisfying stage gate acceptance criteria
  • documentation of procedures

For these work items, who is responsible for designating priorities, assigning to iterations, and iteration planning? The last question is the easiest to answer.  We let the team members doing the work do the task identification, estimating and assignment.  The other questions are a bit more tricky.  I don’t believe that any one person should determine the scope and prioriity of these diverse work items.  Rather, as team lead, I would prefer to let the team leads of infrastructure, customer support and development (traditionally might be PMs) to self-organize and advise me on their priorities and in which iteration their work items should be completed.  The summary of which is then reviewed with the Program Manager to assure alignment with the overall program’s goals.

Therefore what we are doing is co-managing the work item list with myself as facilitator.

This discussion shows some of the challenges in managing a DAD Transition iteration(s) using an agile approach.  I understand that the idea of having shared product ownership is not consist with methods such as Scrum.   However, in my world of large enterprise agile projects pragmatism trumps the agile “code”.  As Barbossa says in Pirates of the Caribbean, “the code is more what you call guidelines than actual rules“.