Author Archives: Scott Ambler

About Scott Ambler

Senior Consulting Partner at SA+A. We help teams to become awesome.

When Does Traditional Software Development Make Sense?

We’re often asked by our customers when it makes sense for a team to take a traditional approach to software development. Sometimes people are honestly trying to identify when each potential lifecycle, including the traditional lifecycle, makes sense to follow. In many cases this request is coming from people who are desperate to continue working in the same old way that they’re comfortable with. They often hope that the context of the situation that they’re in puts them in a position where they don’t need to adopt new ways of working. Some people get “lucky”, although how lucky it is to forgo an opportunity to gain new skills that are currently in demand is questionable at best, but most find that they need to join their colleagues in adopting agile.

Traditional development was prevalent in the 70s through 90s, but starting in the mid-90s organizations started to adopt iterative approaches such as the Unified Process (UP) as well as agile approaches such as Scrum or Extreme Programming (XP). Having said all this, the majority of organizations still have a few teams following traditional lifecycles and will often have people who believe that traditional software development is a good idea. And, as I argue in this blog, in a few situations they’re right about that.

This blog explores several issues surrounding traditional software development:

  1. What is traditional development?
  2. When does it make sense to take a traditional approach?
  3. What factors have no bearing on when traditional makes sense?
  4. But isn’t traditional better at scale?
  5. Does Disciplined Agile (DA) support a traditional lifecycle?
  6. Doesn’t Disciplined Agile Delivery (DAD) have a serial lifecycle?
  7. Why should you listen to me?

 

What is Traditional Development?

Traditional development, also called serial or waterfall, is based on the idea that the delivery lifecycle should be organized into phases or stages. Figure 1 depicts a classic waterfall lifecycle, similar to what Winston Royce originally proposed (and recommended against) in his 1970 paper. In a pure waterfall the flow is unidirectional (from Requirements to Architecture to…) whereas Figure 1 depicts a “modified waterfall” using bi-directional flow that indicates feedback from a phase may go back to the previous phase.

Figure 1. The Waterfall lifecycle.

Figure 2 depicts a gated waterfall where you need to pass through a “quality gate” to move between phases. The “quality gate” tends to be based on the review and acceptance of artifacts – for example a Software Requirements Specification (SRS) coming out of Requirements, a Software Architecture Design (SAD) coming out of Architecture, and so on. It also shows how feedback can go back to any phase via a Change Control process, which on most traditional teams proves to be a change prevention process in practice. I used quotes around “quality gate” because “quality gates” tend to have very little to do with quality in practice and have a lot to do with promoting questionable bureaucracy. My point is that traditional rhetoric may be blinding you to serious deficiencies in this approach.

Figure 2. The Gated Waterfall lifecycle.

Gated Waterfall lifecycle 

Figure 3 depicts the V-model version of the traditional approach where the test phase is depicted as multiple stages and there are clear indications of what each testing activity validates (for example, Integration Test validates your architecture).

Figure 3. The V-model lifecycle.

V lifecycle

The governance strategy with traditional development tends to be artifact based (i.e. someone reviews and signs off on your architecture document) rather than risk based (i.e. we proved that the architecture works by building a vertical slice of the solution that implements high-risk requirements). Each phase tends to be staffed with specialists (i.e. requirements are captured by requirements analysts, the design is created by designers, and so on) which tends to motivate a quality gate approach, the development and maintenance of traceability information, and significant management effort to coordinate everything. This in turn increases cost, overall cycle time and decreases ability to support changing requirements (traditionalists typically fear “scope creep” whereas agilists embrace evolving requirements) which in turn decreases stakeholder satisfaction with the end product.

 

When Does Traditional/Waterfall Make Sense?

There are several situations when the traditional approach makes sense:

  1. Your team has a traditional culture and skillset. Some teams haven’t adopted an agile mindset nor gained agile skills yet, for whatever reason. Teams are more likely to succeed following a strategy that they understand as opposed to one that they don’t.
  2. The project is low risk. The serial nature of the traditional lifecycle makes it poorly suited to address risk, regardless of what adherents of traditional claim. It’s interesting to note that in the original whitepaper describing the traditional lifecycle that Winston Royce pointed out that it wasn’t appropriate for high-risk endeavors, an important tidbit of advice that many organizations have unfortunately ignored over the decades.
  3. You’ve done this sort of thing before. One category of project where traditional seems to thrive are “repeat projects” – you’ve done it before and you know what you’re doing so it’s unlikely that you’ll run into unpleasant surprises. An example of this type of project is the installation of a software package by a company that specializes in doing exactly this. This in effect is a type of low-risk project.
  4. The requirements are unlikely to evolve. There are some projects where the requirements typically don’t change, such as the deployment of a new version of an operating system across many machines (think Windows upgrade) or a straightforward regulatory project (note that some regulatory requirements do evolve in practice, in particular regulations that are “politically charged”). This is also a type of low-risk project.
  5. Transparency isn’t important. Regardless of all the documentation that is generated by a traditional team, the fact is that it’s difficult for senior management to ensure that they have true insight into what is happening on traditional project teams. Another way of saying this is that it’s relatively easy for traditional project managers to mislead you with fanciful status reports and for team members to generate documentation that they believe reviewers are looking for. This lack of transparency is one of the many reasons why traditional teams have a lower success rate on average than agile teams – traditional teams will be in trouble and management won’t know and therefore be unable to guide them out of trouble early on when it is still easy and inexpensive to do so.

 

What Factors Have No Bearing on When Traditional Makes Sense?

My experience is that the following issues do not have an impact, or when they do have very little impact, on the deciding if a traditional strategy is appropriate for your team:

  1. The domain you’re working in. I don’t know of a single domain where agile and lean techniques are not being successfully applied in practice. Not one. We often run into people that say “Sure, agile is fine for e-commerce, but we work in bank so we need to be traditional” or “Sure, agile is fine for banks, but we work in a retailer so we need to be traditional” or “Sure, agile is fine for retailers, but we work for an automobile manufacturer so we need to be traditional” or “Sure, agile is fine for automobile companies but we work for an e-commerce company so we need to be traditional.” But please, don’t take my word for this. Instead do a quick web search on “Agile software development in X” where X is the domain that you think requires a traditional approach – you’ll soon find that many people have figured out how to move beyond traditional in that domain.
  2. The desire to “get it right” the first time. When getting it right the first time the quality focused, tight feedback cycle approaches taken by Disciplined Agilists are far more effective in practice than the heavyweight traditional approaches. When someone tells you that they require a traditional approach to get it right, chances are exceptionally good they know every little about enterprise-class agile development strategies such as DA.
  3. A desire for “predictability.” Agile approaches, due to their greater transparency, are far more predictable than traditional approaches. The heavy documentation and “quality gates” of traditional provide a false sense of predictability to stakeholders who then get frustrated when traditionalists go over budget, are late, drop significant scope to come in on-time and on-budget, or force stakeholders to accept something built to specification instead of something that meets their real needs. You may find the article Examining the Big Requirements Up Front (BRUF) Approach to be an enlightening read.
  4. You’re working at scale. See next section.

 

But Isn’t Traditional Better at Scale?

Not really. Agile and lean are as good or better when applied by large teams, geographically distributed teams, teams in regulatory situations, teams facing domain complexity, teams facing technical complexity, and teams facing organizational distribution. Figure 4 depicts a radar chart of potential tactical scaling factors.

Figure 4. Scaling factors faced by software development teams.

Software Development Tactical Scaling Factors

So how does traditional fair in comparison with agile and lean strategies in practice? If you poke around at the IT Surveys page you’ll see that the 2014 Software Development at Scale study found that agile teams outperformed non-agile teams across a range of success factors. Similar results can also be found in the 2006, 2008, 2010, 2011, and 2013 Project Success Surveys. The Dr. Dobb’s Journal 2008 Project Success study found that when comparing agile and traditional approaches based on level of geographic distribution that agile teams were at least as successful and often more so, on average, than traditional teams. In other words, agile was less risky. In 2006 DDJ found that agile was as good or better than traditional regardless of team size, but unfortunately I can no longer find those results online.

To be clear, some of these surveys are a bit long in the tooth. Having said that, in the past few years organizations have figured out how to successfully apply agile and lean approaches at scale.   I suspect that agile and lean will both have pulled ahead even further compared with traditional at scale. We’re actively looking into these sorts of issues so stayed tuned to this blog for future research results. And, we’re not the only ones who are getting these sorts of results. Go poking around on the web and find out for yourself.

 

Does Disciplined Agile (DA) Support the Traditional Lifecycle?

No, the DA framework does not support a traditional lifecycle. We have purposefully left traditional out of scope for DA.

Having said that, we do recognize that in the vast majority of organizations you will have a multi-modal approach where some teams are following an agile approach, some are lean, some are taking a continuous delivery approach, and some are still following traditional. The more disciplined your organization, the more skilled your people, the less likely it is to have traditional teams in practice.

 

Doesn’t DA Support a Serial Lifecycle?

Yes, but it’s risk-based, not artifact nor activity based as you see with traditional approaches. The two project lifecycles supported by DAD, the Scrum-based Agile lifecycle and the Kanban-based Lean lifecycle, have three phases: Inception, Construction, and Transition. These phases, overviewed in Figure 5, capture the ideas that a project team needs to be initiated somehow (Inception), the solution must be developed (Construction), and then released into Production (Transition).

Figure 5. The phases and milestones of DAD project teams.

When you have stable (long-lived) product teams Inception tends to disappear (with the exception of the need for occasional modeling and planning sessions to think things through) and Transition evolves from a multi-day or multi-week phase into a multi-minute or multi-hour activity. Furthermore, the lifecycle evolves over time as you improve, moving from a phased agile/lean project lifecycle into a continuous delivery lifecycle. More on this in future blogs.

Why Should You Listen to Me?

Here is a brief description of my background in traditional software development:

  1. I spent the late 80s to mid-90s doing traditional development. In fact, two of my books, Process Patterns and More Process Patterns that I wrote in the mid-1990s, describe a CMMI-compliant traditional approach to software development using object-oriented technologies. These books captured my experiences on a range of traditional and quasi-traditional software development projects.
  2. I have significant Unified Process (UP) experience. I have many years of experience with teams following the Unified Process, albeit an iterative methodology rather than traditional one. However, I’ve worked with many organizations that unfortunately instantiated the Unified Process in a traditional manner and then afterwards helped them to recover and evolve their approach into an iterative one and sometimes even an agile one.
  3. Disciplined Agile (DA) leverages traditional strategies. In the DA framework we purposefully leverage strategies from traditional, iterative, agile, and lean sources in recognition that great ideas come from a variety of sources, including traditional ones. In other words, Disciplined Agile is a hybrid framework.
  4. I work with customers still doing traditional today. Almost every client of Scott Ambler + Associates (SA+A) is a multi-modal organization that has some agile teams, some lean teams, some continuous delivery teams, and yes, even some traditional teams. Most of these organizations are looking to improve their overall productivity, which typically means they adopt even more agile and lean strategies and reduce their overall investment in traditional ways of working.
  5. I explicitly research the applicability of various approaches to software development. I have been actively surveying, and sharing all of the results of those surveys publicly in a completely open manner, since the mid-2000s. Several of these research efforts focused on comparing agile, lean, iterative, and traditional approaches to software development in an effort to identify where each approach works well and where they don’t work so well. More on this later as well.

The point is that I believe I have sufficient experience with traditional approaches to speak about where it does and doesn’t work. I hope this blog has provided some valuable insights for you.

 

Agile Adoption and Team Productivity

A common question that people ask is how does the adoption of agile within a team affect its productivity?  The answer to this question will vary by team, but there are several common patterns that we’ve seen over time.  In this blog we explore:

  1. What does increased productivity mean to you?
  2. Agile adoption patterns
  3. What milestones to look out for as a team adopts agile
  4. What you can do to increase your chance of success
  5. How do you know productivity improved?
  6. Agile is about more than productivity improvement

What Does Increased Productivity Mean to You?

Productivity is defines various measures of the efficiency of production, and is calculated as

Value of Output divided by Cost of Input

The implication of this calculation is that there is flexibility in the way that we can increase the productivity of a team:

  • Produce the same output with less cost (i.e. with fewer people).
  • Create greater value with the same number of people.
  • Deliver value incrementally more often, thereby earning value sooner for a longer period of time (this is called decreasing the cost of delay)

Remember that context counts – each team will choose the most appropriate way for them to increase their productivity. Having said that, a common result of a team adopting agile is to incrementally deliver value more often.

Agile Adoption Patterns

The following diagram overviews three common patterns when it comes to productivity change when teams adopt agile.  You’ve likely seen simpler versions of this diagram elsewhere that only show the dark green line, but our experience is that’s only part of the story. You can see in all three cases that the adoption of agile results in an initial productivity loss on a team – this reflects the reality that with any type of change it will take time for a team to learn the new strategy, to identify how it fits into their environment, and to learn the new requisite skills and behaviours. Agile adoption productivity

The three patterns, from least desirable to most desirable, are:

  1. A failed agile adoption (red line). Teams fail to adopt agile for several reasons, usually because the team doesn’t want to adopt agile ways of working, the organization doesn’t properly support their adoption efforts, or the rest of the organization continues to drag them down with traditional ways of working.
  2. A successful agile adoption (green line). Luckily most teams succeed in their agile adoption efforts, and numerous studies have shown a wide range of benefits including faster time to delivery, increased quality, increased stakeholder satisfaction with the delivered solution, and improve team morale to name a few.  Every team is different, but overall on average adopting agile is a positive experience.  You’ll land on this curve when you treat agile adoption as a project, something you do for a few months to help make the team more effective, if you’re successful.
  3. A successful agile adoption evolving into continuous improvement (green dashed line). The most successful teams realize that process improvement isn’t a short-term project but instead is a long-term journey that you undertake.  This is reflected in the dashed line in the diagram below.  You typically start by following a transformation strategy with the team – you get them some initial training, some coaching, help them change their work environment and tooling to be more collaborative.  Then at some point an improvement mindset begins to take hold within the team, one of the fundamental aspects of agility.  The team reflects on a regular basis, identifying potential ways that they can improve and then they experiment with those strategies to see which ones work in practice for them. It’s at this point that they’ve shifted out of the transformation strategy into more of a continuous improvement strategy, which is what enables them to reach higher levels of productivity than what is typically achieved with just a transformation strategy.

What Are the Key Milestones to Look For?

There are three key milestone points on the successful paths that you should watch out for:

  1. Productivity trough (4-8 weeks). With anything new there is always a learning curve, and agile is no exception.  When a team begins to move towards agile their productivity will drop for several reasons: they will likely invest some time taking training, it will take people time to learn new techniques and adopt new ways of working, and it will take time for the team to determine how they will work together following these new agile strategies.  Your productivity levels tend to bottom out after four to eight weeks and then after that will start to improve.  The amount of time varies by team, depending on whether any team members have previous agile experience that they can leverage, how much team members want to change, how effective the training is, and whether you have the support of an experienced and effective coach.
  2. Productivity recovers (2-4 months).  For most teams, the ones who are successful at becoming agile, their productivity levels will recover back to the level they were before they started on their agile improvement journey, within two to four months of starting.  This amount of time depends on the same issues mentioned before.
  3. Improvement culture takes hold (3-6 months). This is the point where the improvement mindset really kicks in and the team starts to explicitly work together to improve the way that they work. This is a reflection that the team is actually “being agile” and not just doing agile ceremonies. Sadly not all teams reach this point and move up onto the dashed green line in the diagram above.  Whether they do so or not is primarily dependent on the willingness of team members to become agile, the quality of the coaching that they receive, and whether your organizational environment allows them to own their process.

How Can You Improve Your Chance of Success?

There are several strategies that you can employ to increase your chance of successfully adopting agile and shifting to a continuous improvement culture within your team:

  1. Invest in training. Get the team started on the right foot with training that not only goes into the fundamental concepts behind agile (“being agile” training) but also works through from beginning to end how agile works in practice (“doing agile” training).  Being agile training is incredibly easy to find, but good “doing agile” training that is comprehensive is much harder to find.  Luckily there are several very good Disciplined Agile training offerings that focus on how to “do agile” in enterprise-class settings.
  2. Hire an experienced coach. A good coach will help your team to avoid common learning pitfalls, and better yet quickly guide you through “learning experiences”, working through with you how to improve the way the team works. Hiring a coach can be a challenge because as we show in Why is it so hard to find qualified coaches? it is possible for anyone, and unfortunately they often do, to claim that they are an agile coach.  Effective coaches have deep experience in what they are coaching as well as skills in the act of coaching itself.  The majority of “agile coaches” tend to to be short on both of these things, and the few coaches that are qualified are in high demand.  There are several Certified Disciplined Agile Partners that you may want to reach out to for Certified Disciplined Agile Coaches (CDACs).
  3. Give the team an “organizational pass.” It’s incredibly difficult for a team to become agile when they are still surrounded by other groups that are actively working in a non-agile manner.  Agile teams need to collaborate with others to achieve their goals (in fact, the 2016 Agility at Scale study found that 96% of people indicated that their team needs to interact with at least one other team).  So, if you want to enable a team to become more agile and improve then you also need to motivate these other teams they rely on – such as the data team, the enterprise architects, and even the governance team – to be sufficiently flexible to work with the agile team in an agile manner.  In some cases the agile team may need to be “given a pass” from creating the mandated artifacts, or having to jump through the mandated “quality gates”, required by these teams.
  4. Help the teams that they collaborate with to become more agile.  The next step after giving an agile team an organizational pass it to recognize that this is an opportunity to experiment with improving other areas within your organization.  Help the enterprise architects to learn about agile and to try a few agile strategies themselves.  Similarly the data team can experiment with agile data management strategies and certainly your IT governance team can also take the opportunity to up their game as well.

How Do You Know That Your Productivity Actually Improved?

Although the chart above intuitively makes sense, how do you know that your productivity has actually increased?  To definitively answer this question you need to determine what productivity means to you, what the productivity level of the team currently is, and then continue to measure the level of productivity over time. This strategy tends to fall apart because few organizations know how they want to measure team productivity and fewer yet have actual measures in place. This of course is particularly vexing when senior management still requires you to prove that your productivity has increased as the result of your agile adoption efforts.  Luckily there are ways to measure the change in productivity even when you don’t know what the baseline productivity level currently is.  We’ll address this topic in a future blog.

Agile is About More Than Productivity Improvement

There are many benefits to agile, improving team productivity being just one of them. Potential benefits, some of which lead to greater productivity, include:

  • Improving the quality of the delivered solution
  • Improved stakeholder satisfaction
  • Greater adaptability to market changes
  • Increased team morale
  • Quicker time to market
  • Greater frequency of delivery
  • and many more

Thank You For Supporting My Ride for Heart

Ride for Heart Medal

On June 4th, 2017 I rode in the Heart & Stroke Ride for Heart event in Toronto.  Together we helped to raise $1,380 (not including any matching funds that occurred during two points during fund raising).  I would like to thank everyone who donated,  they’re acknowledged at the end of this blog, to this great cause.

This was my first year doing the ride and ignoring the horrible weather it was a lot of fun.  The temperature when I started out was 12C and it was pouring rain.  Ugh.  And as you’ll see in one of the pictures below I didn’t have proper rain gear for the ride as I normally don’t ride my bike in the rain (I’m a wimp).  Luckily I signed up for the 25K option, 50K and 75K were the other options.  Last week I had been regretting not signing up for 50K but then given the weather I’m glad I went for the short route.  Next year I’ll do 50K as that goes further up the Don Valley Parkway (DVP) which would be a great ride.

Some Pictures From the Ride

The race started at the Canadian National Exposition (CNE) grounds and we went East along the Gardiner Expressway (the Gardiner and the DVP, the two major highways into downtown Toronto, were closed for the day to allow the ride to occur).  In this picture you can see the Rogers Center (formerly the Skydome) on the left in the downtown core.  This was probably about 2KM into the ride at the time.  If you’re not familiar with Toronto, the Gardiner is a raised highway.

Ride for Heart Gardiner Expressway

This picture was taken around the 20 KM point in the ride.  I’m looking North up the Don Valley Parkway.  As you can see everyone was enjoying the weather, yeah, that’s it.

Ride for Heart DVP

Here I am, back on the Gardiner heading west back to the CNE grounds.  As you can see I was pretty much soaked through and not wearing gloves.  The night before I was going to go out and get some rain gear, but then talked myself out of it thinking that my “water resistant” jacket would suffice.  Great jacket but it was overwhelmed by the amount of rain.  Live and learn.  And I was wearing shorts, which was a good strategy as they got soaked and long pants would have been worse.

Ride for heart Toronto

Acknowledgements

I sincerely thank the following people for their generous donations: Karen Lewis, Andres Alarcon, Antonio Valle Salas, Mike Edwards, Robert Wen, James Densmore, David Wight, Kiron Bondale, Adriano Tavares, Timothy Morris, Glen Little, Jane McGrath, Loreen Ambler, Ron Favali, Olivier Gourment, Sheri Crawford, Terry Hamilton, Kristen Morton, Kevin Brennan, Mike Beedle, Chris Kolde, Renato Putini, Gregg Little, Ellen Grove, Carol McConnell, and Jennifer Yang.

As promised, we’ll be sending out signed copies of our forthcoming book “The Executive’s Guide to Disciplined Agile” to everyone who donated.  This should happen towards the end of July.

Looking forward to next year’s ride!

 

Strategies for Tracking Time on Agile Teams

Time Tracking

In Time Tracking and Agile Software Development we overviewed why teams should consider tracking their time.  Primary reasons include:

  • You’re billing your customer by the hour
  • Your organization wants to account for CapEx/OpEx
  • Your organization wants to take advantage of tax credits (typically for R&D work)

A secondary reason to track time is because the team wants to measure where they are spending time so as to target potential areas to improve.  This is more of a side benefit than anything else – if this was your only reason to track time you’d be better off simply discussing these sorts of issues in your retrospectives.  But if you were already tracking time then running a quick report to provide the team with intel likely makes sense for you.

So what are your options for recording time?  Potential strategies, which are compared in the following table, include:

  1. Automated report from an agile management tool. The basic idea is to extract data from an agile management tool (JIRA, TFS, LeanKit, …) and load it into your time tracking system.
  2. Manual input by team members. Each team member, typically once a week, inputs their time into the time tracking tool.
  3. Manual input by the Team Lead. The Team Lead (ScrumMaster) inputs the time for their team into the time tracking tool.
  4. Manual input by a Project Manager/Coordinator. A PM or Project Coordinator, often in a support role to the team, inputs the time of team.
  5. Don’t track time at all. ‘Nuff said.

Table: Comparing options for tracking time.

Strategy Advantages Disadvantages
Automated report from agile management tool
  • Very efficient because it doesn’t require ongoing data input
  • Sufficient for CapEx/OpEx purposes
  • Sufficient for customer billing when the billing units are by the day (or greater)
  • Requires a bit of development work to feed data from your agile management tool into your time tracking system
  • May motivate the team to start treating the agile management tool like a time tracking tool (which often negates the value of the management tool)
  • Often requires a bit of (programmatic) fudging of the data to calculate the time not captured in the tool (such as coordination meetings, demos, retrospectives, …)
  • May require a bit of negotiation with your organization’s auditors (if any)
  • Only an option for teams using agile management tools
  • Works well for teams that are working in a fairly consistent manner (i.e. mature teams that have gelled)
Manual input by team members
  • Potentially the most accurate approach
  • Sufficient for CapEx/OpEx, tax credits, and customer billing
  • Team members often perceive this as an overhead
  • People will be motivated to input what they believe management wants, particularly if any sort of rewards or punishments are thought to be connected
  • Potential for significant expense across the organization (a few minutes per person per week starts to add up) if this gets too detailed or complicated
  • For people working on multiple teams (a question idea anyway) time tracking often becomes onerous
Manual input by Team Lead
  • Shifts the data input burden away from the team
  • Sufficient for CapEx/OpEx and tax credits
  • Likely sufficient for customer billing
  • Not as accurate as other strategies
  • Takes the Team Lead away from leadership tasks
  • Requires the Team Lead to know what is going on within the team (which frankly should be a given)
Manual input by Project Manager/Coordinator
  • Same as manual input by Team Lead
  • Not as accurate as other strategies
  • Likely requires the PM to interview/badger team members to find out what they did during the week
  • Little better than “make work” for the PM
Don’t track time at all
  • No overhead for the team
  • Your organization may be losing out on tax credits

This blog posting was motivated by a conversation that I had with Stacey Vetzal on Twitter.

Related Reading

How Does Data Management Fit In?

Key tenets of agile and lean are to work collaboratively and to streamline your workflow respectively. This includes all aspects of your workflow, not just the fun software development stuff that we all like to focus on.  This blog posting explores how Data Management activities fit into your overall process.

In the process flow diagram below we see that Data Management is a collaborative effort that has interdependencies with other DA process blades and the solution delivery teams that Data Management is meant to support. Due to the shortened feedback cycles and collaborative nature of the work this can be very different than the current traditional strategies. For example, with a DA approach, the Data Management team works collaboratively with the delivery teams, Operations, and Release Management to evolve data sources. The delivery teams do the majority of the work to develop and evolve the data sources, with support and guidance coming from Data Management. The delivery teams follows guidance from Release Management to add the database changes into their automated deployment scripts, getting help from Operations if needed to resolve any operational challenges. Evolution of data sources is a key aspect of Disciplined DevOps.

Data Management external workflow

This highly collaborative strategy is very different than the typical traditional strategy that requires delivery teams to first document potential database updates, have the updates reviewed by Data Management, then do the work to implement the updates, then have this work reviewed and accepted, then work through your organizations Release Management process to deploy into production.

In the next blog posting in this series we will explore the internal workflow of a Disciplined Agile approach to Data Management.  Stay tuned!

Can You Spare a Few Moments to Fill in Our Short Agile Survey?

Trust Data Not Lore

For those of you who are Star Trek fans you’ve likely been seeing ads for this t-shirt on your social media feeds.  It is an apt metaphor for the empirical approach that we take with Disciplined Agile – we regularly run studies to explore what is actually going on out there on agile teams, we gather data, as opposed to pouting some of the wishful thinking (spreading lore) that we often hear from consultants and vendors.

We are currently running an agile mini-survey of only 6 questions, so this will take you a few minutes at most to fill out, exploring some important issues about agile adoption within your organizations.  We hope that you choose to invest a few minutes of your valuable time to fill it out, and thank you in advance for doing so.

What Will We Do With the Results?

As you already know the surveys that we run are completely open – We share the source data (without identifying information), the questions as they were originally asked, and a Powerpoint deck summarizing our interesting findings after the survey has closed.  In fact, we have the results from dozens of previous studies posted at the IT Surveys page for you to take a look at.

We also write blogs discussing the results.  For example, for the 2016 Agile Scaling survey that we ran in November, we published several blogs:

Recently, we’ve created a new infographic summarizing the results of the study.  If you click on the thumbnail below it will take you to the page where you can download a high-resolution PDF of it.  This infographic is only available to members of the Disciplined Agile Consortium (DAC).

Agility at Scale 2016 Infographic

 

Where Can You Get the T-Shirt?

If you’re interested in the T-Shirt, it is a time limited offering on Teespring.

Why is it so hard to find qualified agile coaches?

Questioning peopleI was hoping to come up with a pithy, short answer to this question but the only thing that I can come up with is “people.”  The not-so-pithy answer is that there is no sort of agreement around what it means to be a “qualified agile coach”, the people hiring coaches aren’t thinking things through in many cases, and the agile community suffers from a myriad of integrity challenges when it comes to professionalism. In this blog I work through the following ideas:

  1. Why is there a dearth of qualified agile coaches?
  2. Sports coaches as an example
  3. What should we expect from agile coaches?
  4. Our solution: Certified Disciplined Agile Coaches (CDACs)
  5. Parting thoughts

Why is There A Dearth of Qualified Agile Coaches?

Let’s answer this in two parts: Why is there a dearth of agile coaches and why are there so many unqualified coaches available?  The first question is very easy to answer.  The demand for agile coaches far outstrips the supply.  The adoption rate for agile has been growing steadily since 2001, hence the growing demand.  As you’ll see later in this blog, it takes years to grow good coaches.  As a result there is little hope for the supply to catch up with demand any time soon.

The second question, why are there so many unqualified coaches available, is easy but uncomfortable to answer.  In general we have systemic challenges in the IT industry and in many ways we’ve managed to exacerbate these problems within the agile community. Some of the challenges within the IT community include:

  • A person is just as likely to be a self taught programmer, and more likely in fact, than they are to have a computer science or engineering degree
  • Although we throw the term “software engineering” around a lot, there is no agreement around what it means or even if it’s an appropriate concept (the IEEE/ACM promotes one, but there are many others)
  • There is no sort of apprenticeship culture in this industry
  • Few people have a personal goal of mastery, and few organizations support the gaining of mastery amongst their staff
  • There is a shortage of talented people, so It is very easy to find and retain employment regardless of your level of talent, and the market for IT people is still growing
  • No country has a licensing body for software professionals that is commonly required by organizations, unlike other professions such as doctors, lawyers, engineers, architects, and many more
  • Many people in the IT community believe this normal for a professional, or if they do perceive a problem they are (rightfully) overwhelmed with the challenge of addressing it
  • Colleges and universities are endemically years behind the quickly changing IT industry (there are a handful of schools that work closely with industry, so it’s getting better)

Then we have the agile community, with its various certification training scams.  You can become a certified master after staying awake during a two-day workshop and passing an online test that almost nobody fails.  To put this into context, a Starbucks barista, the kid who pours your morning coffee, get’s three days of training before being let loose on customers.  Yet it somehow makes sense that someone with 50% less training becomes the lead of a software development team?  Really?  Another example: Someone can become a scaling program consultant after attending a four-day workshop, and worse yet are now “qualified” to teach a two-day workshop to others so as to impart their vast agile scaling knowledge upon them.  Amazingly, because of the demand by companies desperate to hire agile-skilled people, the demand for these “designations” is incredible (shameful would be a more appropriate word).

In practice many agile designations are little more than “participation ribbons”, yet most organizations take them seriously often either because they don’t realize how trivial they are to earn or because they’ve given up expecting any better from agilists.  Is it any surprise that it’s hard to find qualified coaches when we’ve watered things down so much?

Sports Coaches as an Example

CoachCoaching is very common in sports and with the exception of “pick up” games few sports teams are without a coach.  In fact, serious sports teams tend to have several coaches, typically lead by a head coach. In professional sports coaches are paid significant salaries, sometimes millions of dollars a year, as coaching is perceived to be a critical success factor.  It makes sense to look at sports coaching works to see how agile coaching might work.

Most sports coaches are former players.  They’ve typically played for years, and sometimes decades, having been coached themselves all along the way.  They’ll often start off as children, in Canada it’s common for kids to start learning to skate and play hockey at the age of two, being coached and drilled in basic skills and knowledge for years.  They also gain practical experience playing games.  Most kids drop out eventually, although many still play their sport (be it hockey, football, cricket, baseball, …) well into middle age.  And some decide to stay in the sport, but make the shift from being a player into being a coach.

The transition to becoming a sports coach generally isn’t easy.  There are three common strategies for this:

  1. Formal training.  One path is to go to university, get a teaching degree, and become a gym teacher at a middle school or high school and begin coaching children.  These coaches tend to coach a wide range of sports, although in some cases you’ll often see a coach specialize on a single sport, such as high school football or hockey, because that’s what their passion was a child (and often because they hope to move up the ranks at eventually, see point #3 below).
  2. Informal apprenticing.  Another path is to apprentice, asking your existing coach to allow you opportunities to coach others under their guidance.  When I trained in karate this was very common, with senior students helping to teach less senior students.  My daughter is currently learning to skate and they follow a similar pattern with senior skating coaches (adults) being helped by assistants (typically teenagers who have been skating for many years) to coach and teach the younger children.  Helping to teach or coach others is recognized as an important part of your own learning process.
  3. Formal apprenticing.  Because of the money involved professional sports teams tend to take a more formal approach to coaching.  They will often expect coaches either come up through the coaching ranks – start as a high school coach, then become a college-level coach, then finally a professional coach – or to come up through the professional sports ranks – when your star players are past their peak they sometimes move into coaching roles.  Each time you move up to a new level of coaching, say from high school football to college football, you often start as an assistant coach to first prove yourself.  The head coach at each level recognizes that it’s their responsibility to grow more coaches, so they impart their skills and knowledge on to the junior coaches.

So, what are some important observations we can make out of all of this?  First, sports coaches have deep skills and experience at the sports that they are coaching.  Second, we expect this of them.  Would you pay to have your child to be given skating lessons by a “Certified Skating Master” who had two days of training in the “skating mindset” and how to facilitate a handful of skating meetings?  Of course you wouldn’t.  Instead you’d want someone who had been skating for years, and better yet may have even been a competitor at some point in the past.  Third, it takes years of apprenticing or training to become a good sports coach, not just several days in a certification workshop.

What Should We Expect From Agile Coaches?

Here is what we’ve found to be the critical success factors for agile coaches:

  1. They should have years of agile experience, not days of training.  If someone doesn’t have years of experience in something, and more importantly years of varied experience, then why they heck would you hire them as a coach?
  2. They should have coaching skills and experience.  Being experienced in agile isn’t enough.  Apprenticing under another good agile coach is a great way to get coaching skill as is getting training in agile coaching (the Agile Coaching Institute is a start for this although the program at International Coaching Federation (ICF) is far more thorough).  The need for experience is a bit of a catch-22 of course – you need to already be an agile coach to be qualified to be an agile coach.  But, if someone has no previous coaching experience then at best I’d put them into a junior coaching role under the guidance of an experienced coach.  This provides them with the opportunity to gain the requisite experience and to prove themselves in practice.
  3. They should have robust agile skills and knowledge.  Years of agile experience is a good start, but better yet is a range of experience at all aspects of the lifecycle in which they are coaching.  It’s reasonable to expect a delivery team coach to understand all aspects of agile solution delivery so that they can coach the entire team in the skills they need to succeed.  Furthermore, it’s reasonable to expect an Agile IT coach to have experience in the full agile IT lifecycle, including areas such as Enterprise Architecture, Data Management, Portfolio Management, and many others.
  4. They should have experience in a similar context.  Ideally they should have skills in a similar context to what you currently face – someone who only has small team coaching experience will struggle to coach a large programme, someone who only has only coached in startup companies will struggle in a large financial institution, someone who has only coached co-located teams will struggle with globally distributed teams.  Context counts.

Criteria for Effective Agile Coaches
All four of these factors are equally important.  Any “coach” who is deficient in one or more of these areas still has some work to do.  Nobody is perfect of course, given the rates that agile coaches demand it’s reasonable to expect these people to be qualified.

Our Solution: Certified Disciplined Agile Coaches (CDACs)

A fair question to ask is how do we deal with this in the Disciplined Agile (DA) space. We believe that it’s critical to your success to have qualified coaches so we’ve built a principled certification program based on the martial arts philosophy of Shu-Ha-Ri.  Certifications must be earned and that takes time.  The following diagram summarizes our strategy for how practitioners must earn DA certifications.

Agile Practitioner CertificationsTo become a Certified Disciplined Agile Coach (CDAC) you need to have at least five years of experience in agile (which is verified by the Disciplined Agile Consortium (DAC)), plus evidence that you’ve already been sharing your skills and knowledge (we call this give back), plus you must be a Certified Disciplined Agile Practitioner (CDAP).  To become a CDAP you must have at least two years of proven agile experience (validated by DAC), have passed a comprehensive test of your agile knowledge, and already be a Certified Disciplined Agile Practitioner (CDA).  To be a CDA you must have passed a comprehensive test of your knowledge and skills.  So, this process of certification ensures that CDACs have comprehensive skills and knowledge in agile techniques, at least five years experiences in agile, and at least initial experience in coaching/teaching (the giveback component). Note: Not shown in the diagram above is Certified Disciplined Agile Instructor (CDAI), which you must be at least a CDAP and have proven ability to teach DA.

Parting Thoughts

It isn’t easy to find qualified agile coaches, but then again it isn’t impossible either. Our hope is that this blog has provided you with some insight into what you should be looking for in a good agile coach.  Anyone can put a shingle up and say that they’re an “agile coach”, but anyone who wants to say that they are a Certified Disciplined Agile Coach (CDAC) needs to have worked through a rigorous process to earn that qualification.  CDACs have proven knowledge, experience, and give back.  Why settle for less?

Related Reading

Disciplined Agile Values for Data Management

Data Management Mindset

There are several values that are key to your success when transforming to a leaner, more agile approach to Data Management. Taking a cue from the Disciplined Agile Manifesto, we’ve captured these values in the form of X over Y. While both X and Y are important, X proves to be far more important than Y in practice. These values are:

  1. Evolution over definition. The ability to safely and quickly evolve an existing data source, either to extend it to support new functionality or to fix quality issues with it, is absolutely paramount in today’s hyper-competitive environment. Yes, having defined data models and metadata describing them is also important, but nowhere near as important as being able to react to new business opportunities. Luckily agile database techniques, long proven in practice, exist that enable the safe evolution of production data stores.
  2. Holistic organization over Data Management. Earlier we said that data is the lifeblood of your organization. Yes, blood is important but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.” Traditional Data Management approaches often run aground because they locally optimize for data concerns, whereas a DA approach to Data Management recognizes that we must optimize the overall whole. This implies that sometimes we may need to sub-optimize our strategy from a data point of view, for the sake of organizational level optimization.
  3. Sufficiency over perfection. Data sources, like many other IT assets, need to be good enough for the task at hand. The old saw “perfect is the enemy of good” clearly applies in the data realm – too much time has been lost, and opportunities squandered, while development teams were forced to wait on Data Management teams to create (near) perfect models before being allowed to move forward. Traditional data professionals mistakenly assume that production databases are difficult to evolve and as a result strive to get their designs right the first time so as to avoid very painful database changes in the future. The Agile Data method has of course shown this assumption to be wrong, that it is very straightforward and desirable to evolve production databases. A side effect of this revelation is that we no longer need to strive for perfect, detailed models up front. Instead we can do just enough up-front thinking to get going in the right direction and then evolve our implementation (including data sources) over time as our understanding of our stakeholder needs evolve.
  4. Collaboration over documentation. We’ve known for decades that the most effective way to communicate information is face-to-face around a shared sketching environment, and that the least effective is to provide detailed documentation to people. The implication is that we need to refocus our efforts to be more collaborative in nature. As data professionals we need to get actively involved with solution delivery teams: to share our knowledge and skills with those teams, and to enable them to become more effective in working with data. Yes, we will still need to develop and sustain data-related artifacts, but those artifacts should be lightweight and better yet executable in nature.
  5. Cross-functional people over specialized staff. Agilists have come to realize that people are more effective when they are cross-functional (also known as T-skilled or generalizing specialists). Although specialists are very skilled in a narrow aspect of the overall process, the problem is that you need a lot of specialists to perform anything of value and as a result the overall workflow tends to be error prone, slow, and expensive. The other extreme would be to be a generalist, someone who knows a little bit about all aspects of the overall process. But, the challenge with these people is that although they’re good at collaborating with others they don’t actually have the skills to produce concrete value. We need the best of both worlds – a generalizing specialist with one or more specialties so that they can add value AND a general knowledge so that they can collaborate effectively with others and streamline the overall effort.
  6. Automation over manual processes. The only way that we can respond quickly to marketplace opportunities is to automate as much of the bureaucracy as we possibly can. Instead of creating detailed models and documents and then reviewing potential changes against them we capture our detailed specifications in the form of executable tests. This is quickly becoming the norm for specifying both the requirements and designs of code, and the same test-driven techniques are now being applied to data sources. Continuous integration (CI) and continuous deployment (CD) are also being applied to data sources, contributing to improving overall data quality and decreasing the time to safely deploy database updates into production.

As you can see, we’re not talking about your grandfather’s approach to Data Management. Organizations are now shifting from the slow and documentation-heavy bureaucratic strategies of traditional Data Management towards the collaborative, streamlined, and quality-driven agile/lean strategies that focus on enabling others rather than controlling them.

Recommended Reading

Is it Disciplined Agile Delivery (DAD) or Disciplined Agile (DA)?

The quick answer is of course “Yes”.  😉

A couple of years ago we caused a bit of confusion when we expanded the scope of the Disciplined Agile Delivery (DAD) framework to address the activities of an information technology (IT) department.  When we did this we realized that the scope of the framework and the name no longer matched, so we decided to rebrand to be simply the “Disciplined Agile (DA)” framework.  Having said that, sometimes it makes sense to say DAD and sometime DA depending on what you’re focusing on at the time.

The Scope of Disciplined Agile (DA)

As you can see in the following diagram, which depicts the scope of the DA framework, it’s clear why there has been some confusion because the DA framework covers a lot of ground.

Scope of Disciplined Agile

Let’s explore each aspect depicted in the diagram:

  1. Disciplined Agile Delivery (DAD).  DAD addresses all aspects of solution delivery from beginning to end, in a streamlined manner.  This includes initial modelling and planning, forming the team, securing funding, continuous architecture, continuous testing, continuous development, and governance all the way through the lifecycle.  The framework includes support for multiple delivery lifecycles, including but not limited to a basic/agile lifecycle based on Scrum, a lean lifecycle based on Kanban, and a modern agile lifecycle for continuous delivery.
  2. Disciplined DevOpsDisciplined DevOps is the streamlining of IT solution development and IT operations activities, and supporting enterprise-IT activities, to provide more effective outcomes to an organization.
  3. Disciplined Agile IT (DAIT).  As the name suggests DAIT addresses how to apply agile and lean strategies to all aspects of IT.  This includes IT-level activities such as enterprise architecture, data management, portfolio management, IT governance, and other capabilities.
  4. Disciplined Agile Enterprise.  A Disciplined Agile Enterprise is able to anticipate and respond swiftly to changes in the marketplace.  It does this through an organizational culture and structure that facilitates change within the context of the situation that it faces.  Such organizations require a learning mindset in the mainstream business and underlying lean and agile processes to drive innovation.

Some History

The first “1.0 release” was the original Disciplined Agile Delivery book in June of 2012.  As the title suggests the focus was on DAD, although it laid the groundwork for Disciplined DevOps in that it baked in the development side of DevOps right into DAD.  In 2015 we began publishing our work in both Disciplined DevOps and Disciplined Agile IT (DAIT) and renamed the framework to Disciplined Agile (DA) to reflect this expanded scope.  Now, in 2017, we are beginning to flesh out Disciplined Agile Enterprise strategies and will soon begin the share them here on this site.

For the want of a whiteboard….

For the want of a whiteboard the vision was lost,

For the want of a new vision the release was lost,

For the want of a new release the product was lost,

For the want of a new product the customer was lost,

For the want of a new customer the company was lost,

And all for the want of a whiteboard.

With a nod to Benjamin Franklin.