Bridging the Gap Between Agile and Data


At the Agile 2017 conference this week Lynn Winterboer was kind enough to invite me to her workshop which explored how to apply agile strategies in the data space.  She did a great job of facilitating the group of about 40 people through identifying the challenges currently faced by teams. The main issues that the group explored were:

  • Product ownership and data
  • Agile data architecture
  • Data testing and tooling support
  • How to include data people, and activities, in development

Lynn will soon be blogging about the results so I’m not going to dive into that here.  I suspect that her blog post will be very interesting.

What I’d like to do here is share a few thoughts about what I observed:

  1. The discussion was very healthy. This was a group of very smart people coming from different backgrounds.  Everyone was interested in sharing their experiences and working together to solve some tough problems.
  2. Context counts. “Agile and data” is a big topic.  A few people were dealing with the issue of how to address data issues on application development projects, some were focused on agile data warehousing/business intelligence, and some were focused on complex data analytics.  In our conversations it was very clear that strategies which worked for app development wouldn’t work for analytics, and vice versa.  This is why Context Counts is one of Disciplined Agile’s fundamental principles.
  3. The challenges are tough. Everyone was working in organizations that were struggling with these challenges.  For each of issue we could have spent much longer exploring the potential solution(s).
  4. Every challenge we discussed are solved issues.  I’ve always found it frustrating when people, who are very smart and who have been struggling with a problem for awhile, have clearly never thought to simply google “database testing tools” or “agile architecture” to find out what advice is already out there.  When we discussed testing I even asked why people hadn’t done such as search, and then pointed out that there are a lot of tools available and even a few people maintaining lists of such tools (such as 40+ database testing tools).  All of these challenges, and more, have solutions described by techniques of the Agile Data and Agile Modeling methods from about 15 years ago.  These techniques have of course been adopted, and put into context, by the Disciplined Agile (DA) framework and in particular Disciplined Agile Delivery (DAD).
  5. The “factions” behaved differently.  Long ago I pointed out that there’s a cultural impedance mismatch between the data and developer communities, and it was pretty easy to observe during this workshop.  For example, during the workshop we were asked to identify solutions to the challenges.  Lynn wanted us to write this information on flip-chart paper so that she could later scan it and share it with others.  For awhile the groups dominated by data people discussed the solutions without writing anything down.  Their conversations were good, but they quickly got stuck in analysis paralysis mode because they seemed to be afraid to write down information for fear that they couldn’t easily update it (the challenge with having paper to write on instead of whiteboards). The groups dominated by agile people, the ones focused on development and architecture, started by writing on sticky notes and putting them on the flip chart paper.  This fundamental agile modelling strategy enabled them to visualize and share their work transparently, enhancing their conversation and enabling them to move forward easily.
  6. Database (tool) vendors need to up their game. Speaking out tools, database vendors and data warehouse tool vendors really need to up their game.  Here’s a few very harsh questions: Does your database tool vendor or ETL tool vendor offer testing tools that enable both black box and white box database testing?  Answer: very likely no, because their customers don’t demand that of them.  Did your data team even think to ask for such tools?  Answer: very likely not.  How many database testing books do you think have been written? Answer: very few, go do a search and see for yourself.  Why does the data community have such a huge blind spot when it comes to testing? Answer: this is a huge side effect of the cultural impedance mismatch.

I’m very happy to see that Lynn is actively trying to bridge the agile and data communities, helping us to learn from each other.  This is something she’s been doing for years and doing it quite well.  My experience is that both communities would benefit greatly from more collaboration with each other.

How Do You Choose Between Software Lifecycles?

Confused

As you know the Disciplined Agile framework supports several delivery lifecycles – Agile, Lean, Continuous Delivery: Agile, Continuous Delivery: Lean, Exploratory.  We do this because solution delivery teams face different situations, so one lifecycle will not fit all.  This begs the question of when would a team adopt each lifecycle?  That is the focus of this blog.

This blog explores the following topics:

  1. What lifecycles does DAD support?
  2. What criteria should you consider when choosing a lifecycle?
  3. Who chooses?

What Lifecycles Does DAD Support?

The delivery lifecycles we will explore are:

  1. Agile/Basic. DAD’s Agile/Basic lifecycle which extends Scrum’s construction lifecycle. We call this the basic/agile lifecycle because it’s likely where you’re going to start. Common scenarios for adopting this version of the lifecycle include situations where you’re extending Scrum to be sufficient for your needs or you’re transitioning from RUP to a disciplined agile approach.
  2. Lean/Advanced. The Advanced/Lean lifecycle encompasses lean strategies such as minimizing work in process, maximizing flow, a continuous stream of work (instead of fixed iterations), and reducing bottlenecks. New work is pulled from the work item pool when the team has capacity.
  3. Continuous Delivery: Agile.  The Continuous Delivery: Agile lifecycle  is a natural progression from the Agile/Basic lifecycle. Teams typically evolve to this lifecycle from the Agile/Basic lifecycle, often adopting iteration lengths of one-week or less. The key difference between this and the Agile lifecycle is that the continuous delivery lifecycle results in a release of new functionality at the end of each iteration rather than after a set of iterations.
  4. Continuous Delivery: Lean. DAD’s Continuous Delivery: Lean lifecycle is basically a leaner version of the Continuous Delivery: Agile lifecycle where the product is shipped into production or the marketplace on a very regular basis. This could be often as daily, although weekly or monthly is quite common too.
  5. Exploratory/Lean Startup.  DAD’s Exploratory lifecycle is followed by agile or lean teams that find themselves in startup or research situations where their stakeholders have an idea for a new product but they do yet understand what is actually needed by their user base.
  6. Waterfall/Serial.  Although this is not supported by Disciplined Agile, we do recognize that some teams will still follow this more traditional way of working.  For more thoughts on this subject, please see the blog posting When Does Traditional Software Development Make Sense?

 

What Criteria Should You Consider When Choosing a Lifecycle?

The following table compares the lifecycles, suggesting when you would choose to follow each one.

Lifecycle Team Type Time to Market Advantages Disadvantages When to Use
Agile/Basic Project Medium
  • Straightforward lifecycle based on Scrum that is easy to learn due to its prescription
  • Iterations (sprints) motivate teams to build functionality in multi-week batches
  • Releases into production typically months apart
  • Tends to fall apart when requirements change often
Teams new to agile
Lean/Advanced Project Fast
  • Functionality is released into production when it`s ready to go
  • Work can be prioritized via a variety of criteria
  • Small batches of work lead to quick flow
  • Requires greater skill and discipline compared to the Agile lifecycle
Disciplined teams with quickly evolving requirements
Continuous Delivery: Agile Product (long lived) Fast
  • Functionality is released into production regularly at a steady flow (typically weekly)
  • Requires significant skill and discipline
  • Requires automated testing, integration, and deployment
Long-running teams
Continuous Delivery: Lean Product (long lived) Very Fast
  • Functionality is released into production continuously, typically one or more times a day
  • Requires significant skill and discipline
  • Requires automated testing, integration, and deployment
Long-running, disciplined teams
Exploratory/Lean Startup Experimental Fast
  • Quick and inexpensive way to run business experiments
  • Low-risk approach to validating potential new business strategies
  • Requires a way to target a subset of your (potential) customer base
  • Often not applicable in regulatory compliance situations
  • Often perceived as a strategy for startup companies only
Identification of a new product or service offering for the marketplace where there is a high risk of misunderstanding the needs of potential end users
Waterfall/Serial Project Slow
  • Comfortable approach for experienced IT professionals who have not yet transitioned to an agile or lean way of working
  • Tends to be very high-risk in practice due to long feedback cycles and delivery of a solution only at the end of the lifecycle
  • Associated risks are often overlooked by management due to façade of predictability and control provided by the paperwork produced
Low-risk situations where the requirements are stable and the problem has a well-known solution.  For example, upgrading the workstations of a large number of users or migrating an existing system to a new platform

 

Who Should Choose the Lifecycle?

The team.

They will often do this with the guidance of an experienced Disciplined Agile coach, particularly when they are new to DA.  It’s tempting to have your portfolio management team to make this choice, and they may often make a (hopefully solid) suggestion when the first initiated an endeavour, but in the end it needs to be the choice of the team.  As you see in the table above there are common considerations for when to use each lifecycle, but the primary considerations are always the skill and preferences of the team itself.

 

Related Reading

The Seven Disciplined Agile Principles

Principles of the Disciplined Agile Framework

We recently published a collection of articles overviewing the seven principles behind the Disciplined Agile (DA) framework.  The article is entitled Principles Behind the Disciplined Agile Framework and it in turn is supported by detailed articles, one for each principle:

  1. Delight Customers – We delight our customers when our products and services not only fulfill their needs and expectations but surpass them.
  2. Be Awesome – Awesome teams are built around motivated individuals who are given the environment and support required to fulfill their objectives.
  3. Pragmatism – Let’s be as effective as we can be, and that may mean we go beyond being just agile.
  4. Context Counts – Every person, every team, and every organization is unique.  Let’s find and evolve an effective strategy given the situation we actually face.
  5. Choice is Good – Different contexts require different strategies. Teams need to be able to own their own process and to experiment to discover what works in practice for them given the situation that they face. Having process options to choose from, and understanding the trade-offs of those options, enables you to home in on better options sooner.
  6. Optimize Flow – Your organization is a complex adaptive system (CAS) of interacting teams and groups that individually evolve continuously and affect each other as they do. To succeed you must ensure that these teams are well aligned, remained well aligned, and better yet improve their alignment over time.
  7. Enterprise Awareness – When people are enterprise aware they are motivated to consider the overall needs of their organization, to ensure that what they’re doing contributes positively to the goals of the organization and not just to the sub-optimal goals of their team.

We hope that you find these articles insightful.

Announcement: An Executive’s Guide to Disciplined Agile is now available!

An Executive's Guide To Disciplined Agile

We’re happy to announce that our new book, An Executive’s Guide to Disciplined Agile: Winning the Race to Business Agility is now available.  The book is currently available on Amazon in paperback for $39.99 US and will soon be available on the Kindle.

Overview

The agile community has figured out how to build and then continually improve very high-performance software development teams. This is akin to creating a race car engine and then evolving it to get more power, better fuel efficiency, and greater speed. Sadly in many cases we take these great engines, put them into an organizational tractor, and then complain that we’re not winning the race. What we need to do is take our great race car engines (our development teams), put them into a race car (a DevOps ecosystem), have a great pit crew and driver (an effective IT organization), and then provide somewhere to race (an organization that can leverage IT to make money). That’s what this book is all about – Moving from optimizing team performance to optimizing the entire enterprise.

Business agility – being an adaptive, lean, responsive, and learning organization – is the race that enterprises need to win today. Yet there is no quick fix, no silver bullet, to attain business agility. This is a multi-year journey requiring hard work, experimentation, and most importantly a willingness to improve. The Disciplined Agile framework lowers risks and provides a path to accelerate your journey to business agility. The framework is unique in that it is the only one that puts all the pieces together into a cohesive enterprise roadmap for business agility transformation.

This book begins with an overview of the challenges and opportunities that organizations face. We then describe seven principles that provide the underpinnings of the Disciplined Agile framework. Then the book works through Disciplined Agile Delivery (how to build a world-class engine), Disciplined DevOps (the race car), Disciplined Agile IT (the race car and its team), and what it means to be a Disciplined Agile Enterprise (the racing business).  The book ends with a plan for starting with an Agile transformation and then evolving into a long-term continuous improvement strategy.

Do you have the discipline it takes to win the race to business agility?

For More Information

Please visit the page An Executive’s Guide to Disciplined Agile: Winning the Race to Business Agility.

Introducing the Continuous Delivery: Agile Lifecycle

Agile Continuous Delivery Lifecycle

This lifecycle is a natural progression from the Agile/Basic lifecycle. Teams typically evolve to this lifecycle from the Agile/Basic lifecycle, often adopting iteration lengths of one-week or less. The key difference between this and the Agile lifecycle is that the continuous delivery lifecycle results in a release of new functionality at the end of each iteration rather than after a set of iterations. Teams require a mature set of practices around continuous integration and continuous deployment and other Disciplined DevOps strategies. This lifecycle is suitable when:

  • Solutions that can be delivered to stakeholders in a frequent and incremental basis
  • Work remains relatively stable within an iteration
  • Organizations with streamlined deployment practices and procedures
  • Projects where getting value into the hands of stakeholders rapidly, before the entire solution is complete, is critical
  • Teams have mature DevOps practices in place including continuous integration, continuous deployment, and automated regression testing
  • The team is long-lived (stable), working on a series of releases over time

Related Reading

Teal is the New Black

Just as your process must be flexible and adaptive, so must your organization. In Reinventing Organizations Frederick Laloux works through the history, and arguably a maturity model, for organizational design. The premise, which is overviewed in the diagram below (you can click on it for a high-res version), is that over time we’re seeing organizations evolve from tribal and often violent structures (Red) through more formalized hierarchical structures (Amber/Orange) to agile approaches (Green/Teal). Today the vast majority of organizations, believed to be 80-90%, are somewhere on the Amber through Green scale.

Laloux Teal Organizations

There are several important observations we’d like to make about Laloux’s organizational maturity scale:

  1. For your organization to support agile it should at least be (mostly) Green, with a participative and values-based culture, or better yet Teal with a truly adaptive and aware strategy (as we’ve been preaching throughout this chapter).
  2. Your organization will start its improvement journey wherever it currently is on the scale. Laloux’s model provides insights into what your current challenges are likely to be and what potential improvements you should consider making.
  3. Teams can still be agile within Orange and Amber organizations, but the organization itself will struggle with agility due to cultural impediments.
  4. It is difficult to jump organizational levels – in other words if your organization is currently Amber then you need to move through Orange, then Green, and finally Teal.
  5. To move between levels you need the both top-down support from leadership as well as bottom-up support from front-line staff – bottom up, “stealth” improvement efforts will fail on their own when organizational antibodies fight back.

You Want to be At Least Green

Why does your organization need to be at least Green or Teal to become agile? Green organizations support a participative culture style that enables collaboration within and between teams. Green organizations explicitly align people around common values, so injecting any missing agile and lean philosophies often proves to be straightforward. Teal organizations go one step further and bring it all together by explicitly recognizing that they are complex adaptive systems (CASs). This provides an environment where agile teams are able to experiment, learn, collaborate, and most importantly thrive as they find new ways to delight their customers.

Improving Horizontally is Much More Realistic

Laloux himself is very clear about the importance of top-down support if you want to become a Teal organization, or frankly just to move up the hierarchy (say from Red to Amber).  In chapter 3 of Reinventing Organizations he states that for an organization to become Teal that it requires the support of both the CEO/founder and the owners of the company.  Our experience is that a third factor is required, the support of the front-line staff (and better yet middle management), for your transformation to be successful.

Laloux believes that it’s much easier for organizations to improve horizontally – become the best Orange or Amber organization that you can be.  In many ways this is much more conducive to a lean continuous improvement strategy than an “agile transformation” strategy.

Your Organization is Probably a Rainbow

It’s attractive to think that your organizational culture is consistent across the entire enterprise, but it is far more likely that you have teams or divisions with differing color ratings according to Laloux’s model. This is because the culture of a team (or division) is greatly influenced by the leader of that team, and leaders vary in their style, and because teams face unique situations – sometimes a red strategy is the most appropriate given what the team faces. Context counts!

Suggested Reading

When Does Traditional Software Development Make Sense?

We’re often asked by our customers when it makes sense for a team to take a traditional approach to software development. Sometimes people are honestly trying to identify when each potential lifecycle, including the traditional lifecycle, makes sense to follow. In many cases this request is coming from people who are desperate to continue working in the same old way that they’re comfortable with. They often hope that the context of the situation that they’re in puts them in a position where they don’t need to adopt new ways of working. Some people get “lucky”, although how lucky it is to forgo an opportunity to gain new skills that are currently in demand is questionable at best, but most find that they need to join their colleagues in adopting agile.

Traditional development was prevalent in the 70s through 90s, but starting in the mid-90s organizations started to adopt iterative approaches such as the Unified Process (UP) as well as agile approaches such as Scrum or Extreme Programming (XP). Having said all this, the majority of organizations still have a few teams following traditional lifecycles and will often have people who believe that traditional software development is a good idea. And, as I argue in this blog, in a few situations they’re right about that.

This blog explores several issues surrounding traditional software development:

  1. What is traditional development?
  2. When does it make sense to take a traditional approach?
  3. What factors have no bearing on when traditional makes sense?
  4. But isn’t traditional better at scale?
  5. Does Disciplined Agile (DA) support a traditional lifecycle?
  6. Doesn’t Disciplined Agile Delivery (DAD) have a serial lifecycle?
  7. Why should you listen to me?

 

What is Traditional Development?

Traditional development, also called serial or waterfall, is based on the idea that the delivery lifecycle should be organized into phases or stages. Figure 1 depicts a classic waterfall lifecycle, similar to what Winston Royce originally proposed (and recommended against) in his 1970 paper. In a pure waterfall the flow is unidirectional (from Requirements to Architecture to…) whereas Figure 1 depicts a “modified waterfall” using bi-directional flow that indicates feedback from a phase may go back to the previous phase.

Figure 1. The Waterfall lifecycle.

Figure 2 depicts a gated waterfall where you need to pass through a “quality gate” to move between phases. The “quality gate” tends to be based on the review and acceptance of artifacts – for example a Software Requirements Specification (SRS) coming out of Requirements, a Software Architecture Design (SAD) coming out of Architecture, and so on. It also shows how feedback can go back to any phase via a Change Control process, which on most traditional teams proves to be a change prevention process in practice. I used quotes around “quality gate” because “quality gates” tend to have very little to do with quality in practice and have a lot to do with promoting questionable bureaucracy. My point is that traditional rhetoric may be blinding you to serious deficiencies in this approach.

Figure 2. The Gated Waterfall lifecycle.

Gated Waterfall lifecycle 

Figure 3 depicts the V-model version of the traditional approach where the test phase is depicted as multiple stages and there are clear indications of what each testing activity validates (for example, Integration Test validates your architecture).

Figure 3. The V-model lifecycle.

V lifecycle

The governance strategy with traditional development tends to be artifact based (i.e. someone reviews and signs off on your architecture document) rather than risk based (i.e. we proved that the architecture works by building a vertical slice of the solution that implements high-risk requirements). Each phase tends to be staffed with specialists (i.e. requirements are captured by requirements analysts, the design is created by designers, and so on) which tends to motivate a quality gate approach, the development and maintenance of traceability information, and significant management effort to coordinate everything. This in turn increases cost, overall cycle time and decreases ability to support changing requirements (traditionalists typically fear “scope creep” whereas agilists embrace evolving requirements) which in turn decreases stakeholder satisfaction with the end product.

 

When Does Traditional/Waterfall Make Sense?

There are several situations when the traditional approach makes sense:

  1. Your team has a traditional culture and skillset. Some teams haven’t adopted an agile mindset nor gained agile skills yet, for whatever reason. Teams are more likely to succeed following a strategy that they understand as opposed to one that they don’t.
  2. The project is low risk. The serial nature of the traditional lifecycle makes it poorly suited to address risk, regardless of what adherents of traditional claim. It’s interesting to note that in the original whitepaper describing the traditional lifecycle that Winston Royce pointed out that it wasn’t appropriate for high-risk endeavors, an important tidbit of advice that many organizations have unfortunately ignored over the decades.
  3. You’ve done this sort of thing before. One category of project where traditional seems to thrive are “repeat projects” – you’ve done it before and you know what you’re doing so it’s unlikely that you’ll run into unpleasant surprises. An example of this type of project is the installation of a software package by a company that specializes in doing exactly this. This in effect is a type of low-risk project.
  4. The requirements are unlikely to evolve. There are some projects where the requirements typically don’t change, such as the deployment of a new version of an operating system across many machines (think Windows upgrade) or a straightforward regulatory project (note that some regulatory requirements do evolve in practice, in particular regulations that are “politically charged”). This is also a type of low-risk project.
  5. Transparency isn’t important. Regardless of all the documentation that is generated by a traditional team, the fact is that it’s difficult for senior management to ensure that they have true insight into what is happening on traditional project teams. Another way of saying this is that it’s relatively easy for traditional project managers to mislead you with fanciful status reports and for team members to generate documentation that they believe reviewers are looking for. This lack of transparency is one of the many reasons why traditional teams have a lower success rate on average than agile teams – traditional teams will be in trouble and management won’t know and therefore be unable to guide them out of trouble early on when it is still easy and inexpensive to do so.

 

What Factors Have No Bearing on When Traditional Makes Sense?

My experience is that the following issues do not have an impact, or when they do have very little impact, on the deciding if a traditional strategy is appropriate for your team:

  1. The domain you’re working in. I don’t know of a single domain where agile and lean techniques are not being successfully applied in practice. Not one. We often run into people that say “Sure, agile is fine for e-commerce, but we work in bank so we need to be traditional” or “Sure, agile is fine for banks, but we work in a retailer so we need to be traditional” or “Sure, agile is fine for retailers, but we work for an automobile manufacturer so we need to be traditional” or “Sure, agile is fine for automobile companies but we work for an e-commerce company so we need to be traditional.” But please, don’t take my word for this. Instead do a quick web search on “Agile software development in X” where X is the domain that you think requires a traditional approach – you’ll soon find that many people have figured out how to move beyond traditional in that domain.
  2. The desire to “get it right” the first time. When getting it right the first time the quality focused, tight feedback cycle approaches taken by Disciplined Agilists are far more effective in practice than the heavyweight traditional approaches. When someone tells you that they require a traditional approach to get it right, chances are exceptionally good they know every little about enterprise-class agile development strategies such as DA.
  3. A desire for “predictability.” Agile approaches, due to their greater transparency, are far more predictable than traditional approaches. The heavy documentation and “quality gates” of traditional provide a false sense of predictability to stakeholders who then get frustrated when traditionalists go over budget, are late, drop significant scope to come in on-time and on-budget, or force stakeholders to accept something built to specification instead of something that meets their real needs. You may find the article Examining the Big Requirements Up Front (BRUF) Approach to be an enlightening read.
  4. You’re working at scale. See next section.

 

But Isn’t Traditional Better at Scale?

Not really. Agile and lean are as good or better when applied by large teams, geographically distributed teams, teams in regulatory situations, teams facing domain complexity, teams facing technical complexity, and teams facing organizational distribution. Figure 4 depicts a radar chart of potential tactical scaling factors.

Figure 4. Scaling factors faced by software development teams.

Software Development Tactical Scaling Factors

So how does traditional fair in comparison with agile and lean strategies in practice? If you poke around at the IT Surveys page you’ll see that the 2014 Software Development at Scale study found that agile teams outperformed non-agile teams across a range of success factors. Similar results can also be found in the 2006, 2008, 2010, 2011, and 2013 Project Success Surveys. The Dr. Dobb’s Journal 2008 Project Success study found that when comparing agile and traditional approaches based on level of geographic distribution that agile teams were at least as successful and often more so, on average, than traditional teams. In other words, agile was less risky. In 2006 DDJ found that agile was as good or better than traditional regardless of team size, but unfortunately I can no longer find those results online.

To be clear, some of these surveys are a bit long in the tooth. Having said that, in the past few years organizations have figured out how to successfully apply agile and lean approaches at scale.   I suspect that agile and lean will both have pulled ahead even further compared with traditional at scale. We’re actively looking into these sorts of issues so stayed tuned to this blog for future research results. And, we’re not the only ones who are getting these sorts of results. Go poking around on the web and find out for yourself.

 

Does Disciplined Agile (DA) Support the Traditional Lifecycle?

No, the DA framework does not support a traditional lifecycle. We have purposefully left traditional out of scope for DA.

Having said that, we do recognize that in the vast majority of organizations you will have a multi-modal approach where some teams are following an agile approach, some are lean, some are taking a continuous delivery approach, and some are still following traditional. The more disciplined your organization, the more skilled your people, the less likely it is to have traditional teams in practice.

 

Doesn’t DA Support a Serial Lifecycle?

Yes, but it’s risk-based, not artifact nor activity based as you see with traditional approaches. The two project lifecycles supported by DAD, the Scrum-based Agile lifecycle and the Kanban-based Lean lifecycle, have three phases: Inception, Construction, and Transition. These phases, overviewed in Figure 5, capture the ideas that a project team needs to be initiated somehow (Inception), the solution must be developed (Construction), and then released into Production (Transition).

Figure 5. The phases and milestones of DAD project teams.

When you have stable (long-lived) product teams Inception tends to disappear (with the exception of the need for occasional modeling and planning sessions to think things through) and Transition evolves from a multi-day or multi-week phase into a multi-minute or multi-hour activity. Furthermore, the lifecycle evolves over time as you improve, moving from a phased agile/lean project lifecycle into a continuous delivery lifecycle. More on this in future blogs.

Why Should You Listen to Me?

Here is a brief description of my background in traditional software development:

  1. I spent the late 80s to mid-90s doing traditional development. In fact, two of my books, Process Patterns and More Process Patterns that I wrote in the mid-1990s, describe a CMMI-compliant traditional approach to software development using object-oriented technologies. These books captured my experiences on a range of traditional and quasi-traditional software development projects.
  2. I have significant Unified Process (UP) experience. I have many years of experience with teams following the Unified Process, albeit an iterative methodology rather than traditional one. However, I’ve worked with many organizations that unfortunately instantiated the Unified Process in a traditional manner and then afterwards helped them to recover and evolve their approach into an iterative one and sometimes even an agile one.
  3. Disciplined Agile (DA) leverages traditional strategies. In the DA framework we purposefully leverage strategies from traditional, iterative, agile, and lean sources in recognition that great ideas come from a variety of sources, including traditional ones. In other words, Disciplined Agile is a hybrid framework.
  4. I work with customers still doing traditional today. Almost every client of Scott Ambler + Associates (SA+A) is a multi-modal organization that has some agile teams, some lean teams, some continuous delivery teams, and yes, even some traditional teams. Most of these organizations are looking to improve their overall productivity, which typically means they adopt even more agile and lean strategies and reduce their overall investment in traditional ways of working.
  5. I explicitly research the applicability of various approaches to software development. I have been actively surveying, and sharing all of the results of those surveys publicly in a completely open manner, since the mid-2000s. Several of these research efforts focused on comparing agile, lean, iterative, and traditional approaches to software development in an effort to identify where each approach works well and where they don’t work so well. More on this later as well.

The point is that I believe I have sufficient experience with traditional approaches to speak about where it does and doesn’t work. I hope this blog has provided some valuable insights for you.

 

Agile Adoption and Team Productivity

A common question that people ask is how does the adoption of agile within a team affect its productivity?  The answer to this question will vary by team, but there are several common patterns that we’ve seen over time.  In this blog we explore:

  1. What does increased productivity mean to you?
  2. Agile adoption patterns
  3. What milestones to look out for as a team adopts agile
  4. What you can do to increase your chance of success
  5. How do you know productivity improved?
  6. Agile is about more than productivity improvement

What Does Increased Productivity Mean to You?

Productivity is defines various measures of the efficiency of production, and is calculated as

Value of Output divided by Cost of Input

The implication of this calculation is that there is flexibility in the way that we can increase the productivity of a team:

  • Produce the same output with less cost (i.e. with fewer people).
  • Create greater value with the same number of people.
  • Deliver value incrementally more often, thereby earning value sooner for a longer period of time (this is called decreasing the cost of delay)

Remember that context counts – each team will choose the most appropriate way for them to increase their productivity. Having said that, a common result of a team adopting agile is to incrementally deliver value more often.

Agile Adoption Patterns

The following diagram overviews three common patterns when it comes to productivity change when teams adopt agile.  You’ve likely seen simpler versions of this diagram elsewhere that only show the dark green line, but our experience is that’s only part of the story. You can see in all three cases that the adoption of agile results in an initial productivity loss on a team – this reflects the reality that with any type of change it will take time for a team to learn the new strategy, to identify how it fits into their environment, and to learn the new requisite skills and behaviours. Agile adoption productivity

The three patterns, from least desirable to most desirable, are:

  1. A failed agile adoption (red line). Teams fail to adopt agile for several reasons, usually because the team doesn’t want to adopt agile ways of working, the organization doesn’t properly support their adoption efforts, or the rest of the organization continues to drag them down with traditional ways of working.
  2. A successful agile adoption (green line). Luckily most teams succeed in their agile adoption efforts, and numerous studies have shown a wide range of benefits including faster time to delivery, increased quality, increased stakeholder satisfaction with the delivered solution, and improve team morale to name a few.  Every team is different, but overall on average adopting agile is a positive experience.  You’ll land on this curve when you treat agile adoption as a project, something you do for a few months to help make the team more effective, if you’re successful.
  3. A successful agile adoption evolving into continuous improvement (green dashed line). The most successful teams realize that process improvement isn’t a short-term project but instead is a long-term journey that you undertake.  This is reflected in the dashed line in the diagram below.  You typically start by following a transformation strategy with the team – you get them some initial training, some coaching, help them change their work environment and tooling to be more collaborative.  Then at some point an improvement mindset begins to take hold within the team, one of the fundamental aspects of agility.  The team reflects on a regular basis, identifying potential ways that they can improve and then they experiment with those strategies to see which ones work in practice for them. It’s at this point that they’ve shifted out of the transformation strategy into more of a continuous improvement strategy, which is what enables them to reach higher levels of productivity than what is typically achieved with just a transformation strategy.

What Are the Key Milestones to Look For?

There are three key milestone points on the successful paths that you should watch out for:

  1. Productivity trough (4-8 weeks). With anything new there is always a learning curve, and agile is no exception.  When a team begins to move towards agile their productivity will drop for several reasons: they will likely invest some time taking training, it will take people time to learn new techniques and adopt new ways of working, and it will take time for the team to determine how they will work together following these new agile strategies.  Your productivity levels tend to bottom out after four to eight weeks and then after that will start to improve.  The amount of time varies by team, depending on whether any team members have previous agile experience that they can leverage, how much team members want to change, how effective the training is, and whether you have the support of an experienced and effective coach.
  2. Productivity recovers (2-4 months).  For most teams, the ones who are successful at becoming agile, their productivity levels will recover back to the level they were before they started on their agile improvement journey, within two to four months of starting.  This amount of time depends on the same issues mentioned before.
  3. Improvement culture takes hold (3-6 months). This is the point where the improvement mindset really kicks in and the team starts to explicitly work together to improve the way that they work. This is a reflection that the team is actually “being agile” and not just doing agile ceremonies. Sadly not all teams reach this point and move up onto the dashed green line in the diagram above.  Whether they do so or not is primarily dependent on the willingness of team members to become agile, the quality of the coaching that they receive, and whether your organizational environment allows them to own their process.

How Can You Improve Your Chance of Success?

There are several strategies that you can employ to increase your chance of successfully adopting agile and shifting to a continuous improvement culture within your team:

  1. Invest in training. Get the team started on the right foot with training that not only goes into the fundamental concepts behind agile (“being agile” training) but also works through from beginning to end how agile works in practice (“doing agile” training).  Being agile training is incredibly easy to find, but good “doing agile” training that is comprehensive is much harder to find.  Luckily there are several very good Disciplined Agile training offerings that focus on how to “do agile” in enterprise-class settings.
  2. Hire an experienced coach. A good coach will help your team to avoid common learning pitfalls, and better yet quickly guide you through “learning experiences”, working through with you how to improve the way the team works. Hiring a coach can be a challenge because as we show in Why is it so hard to find qualified coaches? it is possible for anyone, and unfortunately they often do, to claim that they are an agile coach.  Effective coaches have deep experience in what they are coaching as well as skills in the act of coaching itself.  The majority of “agile coaches” tend to to be short on both of these things, and the few coaches that are qualified are in high demand.  There are several Certified Disciplined Agile Partners that you may want to reach out to for Certified Disciplined Agile Coaches (CDACs).
  3. Give the team an “organizational pass.” It’s incredibly difficult for a team to become agile when they are still surrounded by other groups that are actively working in a non-agile manner.  Agile teams need to collaborate with others to achieve their goals (in fact, the 2016 Agility at Scale study found that 96% of people indicated that their team needs to interact with at least one other team).  So, if you want to enable a team to become more agile and improve then you also need to motivate these other teams they rely on – such as the data team, the enterprise architects, and even the governance team – to be sufficiently flexible to work with the agile team in an agile manner.  In some cases the agile team may need to be “given a pass” from creating the mandated artifacts, or having to jump through the mandated “quality gates”, required by these teams.
  4. Help the teams that they collaborate with to become more agile.  The next step after giving an agile team an organizational pass it to recognize that this is an opportunity to experiment with improving other areas within your organization.  Help the enterprise architects to learn about agile and to try a few agile strategies themselves.  Similarly the data team can experiment with agile data management strategies and certainly your IT governance team can also take the opportunity to up their game as well.

How Do You Know That Your Productivity Actually Improved?

Although the chart above intuitively makes sense, how do you know that your productivity has actually increased?  To definitively answer this question you need to determine what productivity means to you, what the productivity level of the team currently is, and then continue to measure the level of productivity over time. This strategy tends to fall apart because few organizations know how they want to measure team productivity and fewer yet have actual measures in place. This of course is particularly vexing when senior management still requires you to prove that your productivity has increased as the result of your agile adoption efforts.  Luckily there are ways to measure the change in productivity even when you don’t know what the baseline productivity level currently is.  We’ll address this topic in a future blog.

Agile is About More Than Productivity Improvement

There are many benefits to agile, improving team productivity being just one of them. Potential benefits, some of which lead to greater productivity, include:

  • Improving the quality of the delivered solution
  • Improved stakeholder satisfaction
  • Greater adaptability to market changes
  • Increased team morale
  • Quicker time to market
  • Greater frequency of delivery
  • and many more

Thank You For Supporting My Ride for Heart

Ride for Heart Medal

On June 4th, 2017 I rode in the Heart & Stroke Ride for Heart event in Toronto.  Together we helped to raise $1,380 (not including any matching funds that occurred during two points during fund raising).  I would like to thank everyone who donated,  they’re acknowledged at the end of this blog, to this great cause.

This was my first year doing the ride and ignoring the horrible weather it was a lot of fun.  The temperature when I started out was 12C and it was pouring rain.  Ugh.  And as you’ll see in one of the pictures below I didn’t have proper rain gear for the ride as I normally don’t ride my bike in the rain (I’m a wimp).  Luckily I signed up for the 25K option, 50K and 75K were the other options.  Last week I had been regretting not signing up for 50K but then given the weather I’m glad I went for the short route.  Next year I’ll do 50K as that goes further up the Don Valley Parkway (DVP) which would be a great ride.

Some Pictures From the Ride

The race started at the Canadian National Exposition (CNE) grounds and we went East along the Gardiner Expressway (the Gardiner and the DVP, the two major highways into downtown Toronto, were closed for the day to allow the ride to occur).  In this picture you can see the Rogers Center (formerly the Skydome) on the left in the downtown core.  This was probably about 2KM into the ride at the time.  If you’re not familiar with Toronto, the Gardiner is a raised highway.

Ride for Heart Gardiner Expressway

This picture was taken around the 20 KM point in the ride.  I’m looking North up the Don Valley Parkway.  As you can see everyone was enjoying the weather, yeah, that’s it.

Ride for Heart DVP

Here I am, back on the Gardiner heading west back to the CNE grounds.  As you can see I was pretty much soaked through and not wearing gloves.  The night before I was going to go out and get some rain gear, but then talked myself out of it thinking that my “water resistant” jacket would suffice.  Great jacket but it was overwhelmed by the amount of rain.  Live and learn.  And I was wearing shorts, which was a good strategy as they got soaked and long pants would have been worse.

Ride for heart Toronto

Acknowledgements

I sincerely thank the following people for their generous donations: Karen Lewis, Andres Alarcon, Antonio Valle Salas, Mike Edwards, Robert Wen, James Densmore, David Wight, Kiron Bondale, Adriano Tavares, Timothy Morris, Glen Little, Jane McGrath, Loreen Ambler, Ron Favali, Olivier Gourment, Sheri Crawford, Terry Hamilton, Kristen Morton, Kevin Brennan, Mike Beedle, Chris Kolde, Renato Putini, Gregg Little, Ellen Grove, Carol McConnell, and Jennifer Yang.

As promised, we’ll be sending out signed copies of our forthcoming book “The Executive’s Guide to Disciplined Agile” to everyone who donated.  This should happen towards the end of July.

Looking forward to next year’s ride!

 

Strategies for Tracking Time on Agile Teams

Time Tracking

In Time Tracking and Agile Software Development we overviewed why teams should consider tracking their time.  Primary reasons include:

  • You’re billing your customer by the hour
  • Your organization wants to account for CapEx/OpEx
  • Your organization wants to take advantage of tax credits (typically for R&D work)

A secondary reason to track time is because the team wants to measure where they are spending time so as to target potential areas to improve.  This is more of a side benefit than anything else – if this was your only reason to track time you’d be better off simply discussing these sorts of issues in your retrospectives.  But if you were already tracking time then running a quick report to provide the team with intel likely makes sense for you.

So what are your options for recording time?  Potential strategies, which are compared in the following table, include:

  1. Automated report from an agile management tool. The basic idea is to extract data from an agile management tool (JIRA, TFS, LeanKit, …) and load it into your time tracking system.
  2. Manual input by team members. Each team member, typically once a week, inputs their time into the time tracking tool.
  3. Manual input by the Team Lead. The Team Lead (ScrumMaster) inputs the time for their team into the time tracking tool.
  4. Manual input by a Project Manager/Coordinator. A PM or Project Coordinator, often in a support role to the team, inputs the time of team.
  5. Don’t track time at all. ‘Nuff said.

Table: Comparing options for tracking time.

Strategy Advantages Disadvantages
Automated report from agile management tool
  • Very efficient because it doesn’t require ongoing data input
  • Sufficient for CapEx/OpEx purposes
  • Sufficient for customer billing when the billing units are by the day (or greater)
  • Requires a bit of development work to feed data from your agile management tool into your time tracking system
  • May motivate the team to start treating the agile management tool like a time tracking tool (which often negates the value of the management tool)
  • Often requires a bit of (programmatic) fudging of the data to calculate the time not captured in the tool (such as coordination meetings, demos, retrospectives, …)
  • May require a bit of negotiation with your organization’s auditors (if any)
  • Only an option for teams using agile management tools
  • Works well for teams that are working in a fairly consistent manner (i.e. mature teams that have gelled)
Manual input by team members
  • Potentially the most accurate approach
  • Sufficient for CapEx/OpEx, tax credits, and customer billing
  • Team members often perceive this as an overhead
  • People will be motivated to input what they believe management wants, particularly if any sort of rewards or punishments are thought to be connected
  • Potential for significant expense across the organization (a few minutes per person per week starts to add up) if this gets too detailed or complicated
  • For people working on multiple teams (a question idea anyway) time tracking often becomes onerous
Manual input by Team Lead
  • Shifts the data input burden away from the team
  • Sufficient for CapEx/OpEx and tax credits
  • Likely sufficient for customer billing
  • Not as accurate as other strategies
  • Takes the Team Lead away from leadership tasks
  • Requires the Team Lead to know what is going on within the team (which frankly should be a given)
Manual input by Project Manager/Coordinator
  • Same as manual input by Team Lead
  • Not as accurate as other strategies
  • Likely requires the PM to interview/badger team members to find out what they did during the week
  • Little better than “make work” for the PM
Don’t track time at all
  • No overhead for the team
  • Your organization may be losing out on tax credits

This blog posting was motivated by a conversation that I had with Stacey Vetzal on Twitter.

Related Reading