Do Agile Teams Take on Hard Problems?

We often hear that agile software development is fine when you face a simple problem, but that agile isn’t sufficient for more difficult problems.  Of course this falsehood is promoted by people who have little or no agile experience, or who have been involved with a failed “agile adoption” (usually these teams adopted ad hoc strategies thinking they were agile).  Anyway, we decided to look into whether agile teams are taking on hard domain problems in practice.

The following diagram summarizes the responses to our question around agile teams and compliance from our 2016 Agility at Scale study.  As you can see, 40% of respondents indicated that their agile team faced either complex or very complex problems, and that a further 38% faced medium complexity.  Interestingly, only one in eight respondents said that their team faced a straight forward problem.

Agile and Domain Complexity

The bottom line is that agile strategies, and in particular disciplined agile strategies, are in fact applicable for taking on complex problems.  More importantly, this is happening in practice around the world on a regular basis.

Related Posts

Do Agile Teams Face Regulatory Compliance?

We often hear that agile is great for simple situations but as soon as you face compliancy issues that it doesn’t work.  Is it possible to be agile when you face regulatory compliance, such as PCI and FDA compliancy?  Is it possible to be agile when you face organizational compliance, such as working in a CMMI regime?  Important questions that we decided to look into.

The following diagram summarizes the responses to our question around agile teams and compliance from our 2016 Agility at Scale study.  As you can see, 62% of respondents indicated that their agile team faced some form of regulatory compliance, 20% some form of organizational compliance, and 15% said both.  In fact, two-thirds of agile teams operate under one or more compliancy requirements.

Agile Regulatory Compliance

For further reading about compliancy, please read our detailed blog posting Agile and Regulatory Compliance.

Related Posts

How “Whole” Are Agile Teams in Practice?

We are often told that agile teams should be whole, that they should have sufficient people, funding, and skills to fulfill their mission.  The idea is that this reduces the dependencies that your team has on others, enabling them to make decisions and to collaborate more effectively.  But, is this actually happening in practice?  Are agile teams truly whole, or do they still need to collaborate with other teams (hopefully productively) to get the job done?  Being strong believers in empiricism over rhetoric we decided to look into this issue.

In November of 2016 we ran the 2016 Agility at Scale survey.  It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on.  The following infographic summarizes the findings from the question that explored whether agile delivery teams need to work with external teams or groups to get their work done – in other words, are agile teams truly whole or do they rely on others?  As you can see, 96% of respondents indicated that in practice their team had to work with one or more other teams, leading to the conclusion that very few agile teams appear to be truly whole.

Agile Teams Need to be Enterprise Aware

One of the fundamental principles underlying the Disciplined Agile framework is that disciplined agilists should be enterprise aware – they should recognize that they need to collaborate with others outside of their team, that they should work towards a common organizational vision, and that they should strive to do what is best for the organization and not just what is convenient for them.  Given that agile teams are collaborating with others in practice, it is clear that this philosophy of being enterprise aware is important.

The following diagram presents the results from the survey question in greater detail.  You can obtain the source data, a copy of the original questions, and a slide deck key diagrams at the 2016 Agility at Scale survey page.

Enterprise Awareness

Related Posts:

Vertical Slicing for a Data Warehouse (DW)/Business Intelligence (BI) Solution

There are several strategies that you can choose to employ with vertically slicing the requirements for a DW/BI solution. These strategies are described in the following table. There are example stories for each strategy as well as some advice for when to apply each strategy.

Table 1. Vertical slicing strategies for a DW/BI solution.

Slicing Strategy Example Stories When to Do This
One new data element from a single data source
  • As a Professor I would like to know the names of my students so that I know who should be there
  • As a Student I would like to know what courses are taught at the university
Very early days when you are still building out fundamental infrastructure components. Very common for the first iteration or two of Construction. These slices still add real business value, albeit minimal.
One new data element from several sources
  • As a Professor I would like the student list for a seminar that I teach
  • As a Student I would like to know what seminars are being taught this semester
Early days during Construction when you are still building out the infrastructure. These slices add some business value, often fleshing a DW data element to include the full range of data values for it.
A change to an existing report
  • As a Professor I would like to know the standard deviation of marks within a seminar that I teach
  • As a Student I would like to know how many spots are still available in a seminar
Evolution of existing functionality to support new decision making
A new report
  • As a Professor I would like to know the distribution curve of student marks in a seminar that I teach so I may adjust accordingly
  • As a Registrar I would like to know what Seminars are close to being full
Several iterations into Construction when the DW/BI solution has been built up sufficiently.
A new reporting view
  • As a Registrar I would like to know what the prerequisites are for a seminar so that I can advise students
  • As a Professor I would like to know the current course load of each student within a seminar that I teach
Several iterations into Construction when the DW/BI solution has been built up sufficiently.
A new DW/DM table
  • As a Chancellor I would like to track the revenues generated from parking pay meters to identify potential profits to divert to supporting students
  • As a Professor I would like to recommend suggested readings to help people prepare before taking a seminar
Several iterations into Construction when the DW/BI solution has been built up sufficiently.

There are several interesting things about the stories in the table:

  1. They are written from the point of view of your stakeholders. They aren’t a technical specification. For example, the first story describes how professors want a list of student names but it isn’t saying from what data source(s), what the element names are, … These are design issues, not requirement issues.
  2. They always provide business value. The first story appears to be the beginnings of an attendee list for a seminar. Having something as simple as a list of names does in fact provide a bit of value to professors.
  3. Sometimes that business value isn’t (yet) sufficient. It may take several iterations to implement something that your stakeholders want delivered into production, particularly at first. For example, although a list of student names is the beginnings of a class list it might not be enough functionality to justify putting it into production. Perhaps professors also need to know the program that the student is enrolled in, their current year of study, and basic information about the seminar such as the course name, time, and location of it. The decision as to whether the functionality is sufficient to ship is in the hands of your stakeholder (this is one of the reasons why you want to demo your work on a regular basis).

I’ve written a detailed explanation of vertical slicing for a DW/BI solution, and of course there is a wealth of information about agile database techniques in general for those of you interested in greater detail.  You may also find our one-day Disciplined Agile Data Warehousing/Business Intelligence Workshop to be a valuable learning experience too.

How Geographically Distributed Are Agile Teams in Practice?

Many people, particularly those new to agile, will tell you that agile teams must be small and co-located.  That is certainly a smart way of organizing a team, but is isn’t required.  In fact agile teams are more likely to be geographically distributed in some way than they are to be co-located.  In practice, not theory.

In November of 2016 we ran the 2016 Agility at Scale survey.  It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on.  The following graph summarizes the responses around geographic distribution.

Geographic distribution and agile teams

The survey found that less than one-third of agile teams are near-located, where all of the IT members are either co-located or at least in a shared open space.  Previous studies have found that this number drops to one-in-ten teams being near located when you also include primary stakeholders.

Don’t let anyone tell you that you can’t do agile with a geographically distributed team because others are clearly doing so in practice.  Yes, geographically distributed agile is different than near-located agile, which is one of the reasons why you need to take a pragmatic, context-sensitive approach to agile solution delivery.  The Disciplined Agile framework provides the foundation from which to scale your approach to solution delivery to address a range of scaling factors, including geographic distribution.  In fact, you may find our article around geographically distributed agile teams to be an interesting read.

Related Posts:

How Large Are Agile Teams in Practice?

The contrite answer is that they’re as large as they need to be, and the contrite agile answer is that they’re as small as they can be.  Now that we’ve gotten the contrite answers out of the way, how large are agile teams in practice?

In November of 2016 we ran the 2016 Agility at Scale survey.  It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on.  The following graph summarizes the responses around team size.

Agile team size

This year’s survey found that roughly half (48%) of agile teams are more than 10 people in size and one-quarter are more than 20 people in size.  These findings are similar to what we’ve found in the past with both the 2012 Agility at Scale survey and the 2009 Agility at Scale survey.

In short, don’t let anyone tell you that you can’t do agile with a large team because others are clearly doing so in practice.  Yes, large team agile is different than small team agile, which is one of the reasons why you need to take a pragmatic, context-sensitive approach to agile solution delivery.  The Disciplined Agile framework provides the foundation from which to scale your approach to solution delivery to address a range of scaling factors, including team size.  In fact, you may find our article around large agile teams to be of interest.

Related Posts:

 

New goal: Develop Initial Test Strategy

Fire works

We have recently published a new Inception process goal, Develop Initial Test Strategy.  The potential need for this goal was identified a little over a year ago by an organization that we were actively working with and since then we have worked with several other organizations where it was also clear that this process goal was needed.

The basic idea is that during the Inception phase the team should consider developing an initial test strategy so as to identify how they will approach verification and validation. This includes thinking about how you intend to staff your team, organize your team, capture your plan, approach testing, approach development/programming, choose a platform for test environment(s) platform, choose a platform equivalency strategy, test non-functional requirements, automate test suites, obtain test data, automate builds, report defects, and govern your quality efforts. The goal diagram is depicted in Figure 1 and the update goals overview for Disciplined Agile Delivery (DAD) is shown in Figure 2.

Figure 1. The goal diagram for Develop Initial Test Strategy.
Develop Initial Test Strategy

Figure 2. The process goals diagram for DAD.

Process Goals of Discipined Agile Delivery

Update: Secure Funding Goal Diagram

Update

We recently updated the Secure Funding goal diagram, shown below.  The change was the addition of the Choose Funding Scope decision point, to recognize that the type of team you’re funding will affect how you fund said team.  We have also published details about each decision point and its associated options, including the tradeoffs of each, at the Secure Funding overview page.  This is information that we previously had only published in the first Disciplined Agile Delivery book but we’ve decided to start publishing updated versions of this information to the web to make it more accessible to you.

Secure Funding

Agile is supposed to be easy, right?

The Agile manifesto is only 4 lines, there are only 12 principles for agile software and the original Scrum Guide was less than 20 pages, so how hard can Agile be?

 

Chess has only a couple of dozen basic moves so how hard can it be to get to checkmate?

 

 

How many times have you heard?

  • All you need to be Agile is to get the team doing a few funky ceremonies

Or the approach to evolving a team from forming to performing:

  • If the team isn’t performing after a month then the manager is not effective.

Despite the apparent simplicity of Agile, transforming to Agile is hard and the learning curve is very steep.  Depending on the team and the individuals, a transformation from waterfall to a high performing Agile team can take anywhere from 6 months to a year or more.  That’s not to say that they won’t be productive over that period of time.  In fact, they will start delivering business value at the end of the first iteration, but the team will be in constant growth and development over that period of time because not only do the people have to learn a new process, they also need to learn a new way of thinking as well as a new way of working as a team.

1.0 A new process

Aside from being excited to work with the thought leaders in the agile world, the reason I joined with SA+A is because of the Discipline Agile Delivery (DAD) decision framework and the way it captured all the tough stuff that I had been wrestling with for years.  With many years of experience building agile teams, I considered myself somewhat of an expert with answers to a lot of the difficult questions.  When I reviewed the DAD framework I realized that the answers I had gained from my own personal experience would not apply in all circumstances or for all teams.  Context counts.  The DAD Agile/Basic and Lean/Advanced project lifecycles have three phases: inception, construction and transition.  Each phase has a collection of process goals, and each goal encompasses several process decision points.  And, each decision point has a collection of options to choose from.  These process goals, decision points and options have been empirically assembled by analyzing many agile teams to see what makes an agile team successful.  This framework gives a team the ability to define a process that works for them.  Is this difficult to do?  Absolutely!  Defining a process to use is always difficult. BUT…..  The framework provides some guidance. Each goal option has default options that the team can use to get started and the options are listed in preference order so the team can easily pick an initial option.  As the team evolves and matures they can chose the more advanced options and evolve and mature their process.  Does the framework include all possible options?  Definitely not, but the options are certainly extensive.  The framework has evolved over time and continues to evolve as the coaches and thought leaders gain more experience and work with more diverse teams.

No matter which lifecycle the team has chosen to use; whether the team is using classic Scrum (Basic/Agile) or Kanban (Advanced/Lean) or a tailored process created using a framework such as DAD, the team is going to need practice and discipline to get it right.  That means, executing the lifecycle, doing periodic retrospectives, updating the process and then repeating again and again until they get it right.  Changing behaviors is hard and the team will struggle to be successful.

2.0 A new way of thinking

No matter how you cut it, transforming to Agile is a cultural change and culture changes are hard.  Not only is it hard for those who want to make the change but almost inevitably there will be pockets of resistance from people are happy with “the way we’ve always done it”.  Unfortunately, political infighting is the biggest risk to innovation. It is going to happen, you may as well brace yourself for it.

The roles are changing from traditional roles (such as: developer, business analyst (BA), system analyst (SA), quality assurance (QA) and project manager (PM), to new unfamiliar roles (such as: product owner (PO), team lead (TL), architecture owner (AO) and team member).  The traditional roles may be deeply embedded in your HR hiring and performance review processes. Transitioning to the new roles will require a realignment of skills and abilities that is bound to cause great consternation amongst the team members.  Often this is a very difficult transition for project managers, in particular, because they find that a lot of the traditional tasks (i.e. work breakdown structures, planning, estimating, etc.) are now the responsibility of the whole team.

The team needs to start thinking in smaller deliverable pieces.  The concept of delivering a complete system from known specifications at the beginning of the project has been the fallacy of traditional development.  The cost of delaying delivery to the very end of the project and the lack of ability to meet actual customer needs can be eliminated by delivering minimum functionality early and augmenting with new and improved functionality iteration after iteration in close collaboration with the end customer.  But moving to this model of continuously delivering can be a tough adjustment for developers.  Even (or maybe especially) for seasoned developers, the concept of starting to build and deliver not knowing all the detailed specifications up front can be very unsettling. Getting comfortable with this model takes time and experience before developers cease to fall back on old behaviors.

One of the pillars of agile is complete transparency with the business and the stakeholders into the team, the process they are using and the delivery of the business functionality.  With the business embedded in the team on a day to day basis and iteration reviews to demonstration and approve all the deliverables from each iteration, there is no hiding problems when they occur and no hiding late deliveries or failed iterations.  On the flip side, the business is there to celebrate the wins and recognize the deliveries at the end of each iterations and they are part of determining the velocity of the team and adjusting the schedule to match the velocity.

The new way of thinking also requires new tools.  Tools that can build dashboards to automate and display the development intelligence metrics that provide transparency into the team’s performance.  Tools that automate configuration management, continuous integration and continuous integration as well as automated testing.  All of these endeavors are hard and required skilled individuals to lead; none of these comes for free.

3. A new way of working with others

More people skills are required by everyone.  People are now going to be working much closer together physically and will be interacting and collaborating on a much more frequent basis.  Experience working together (often for many years) may give an advantage to some team members, however, they still need to learn how to work together as a team.

3.1 Team Development Phases

Tuckman’s team development model includes the phases: forming, storming, norming, and performing. These phases are all necessary and inevitable in order for the team to grow, to face up to challenges, to tackle problems, to find solutions, to plan work, and to deliver results.  The number of iterations required to move through the phases will vary by team depending on factors such as: company culture, skills, experience, familiarity with each other, leadership (or lack thereof), external influences and coaching etc. Teams may also plateau in a phase before entering the next phase.

Agile teams will go through the four Tuckman phases:

  1. Forming. All teams will go through the forming phase. Typically this take 4 to 6 iterations for the team to: understand the lifecycle process, understand how to work together, understand what the team goal is, and most importantly to get comfortable enough with each other to risk the possibility of conflict.  During this phase, the team’s velocity will vary considerably especially for the expected failed iterations although there should generally be a trend upward in the velocity.  Retrospectives in this phase tend to be very cordial and factual because everyone is avoiding conflict.
  2. Storming. The storming phase can be very upsetting to both the team members and team managers. Often managers will step in to solve the storming issues without realizing that it is a necessary phase for the team to work through on their own.  Conflict, disagreements and personality clashes must get resolved before the team can move to the next phase.  Unfortunately, this phase is a make or break phase and it is hard to predict just how many iterations it will take but it is not usual to expect 3 or 4 pretty rough iterations.  Some teams will completely disintegrate during this phase and it is certainly not unusual to have a decrease in velocity during this phase.  Some members of the team may find they have basic incompatibilities and need to move to another team and this change in team membership can revert the team back to the forming phase.  The retrospectives tend to be very interesting and intense during this phase as they deal with the inter-personal issues of the team.  You can pretty much expect that there are going to be some hurt feelings no matter how safe an environment you have.
  3. Norming. The norming phase is when the team comes back together stronger and more cohesive for having weathered the storming phase. They have resolved their issues and have developed a new synergy where the team just “works well together” and it is fun to be part of the team. The team velocity should trend upward and then level off as the team hits its capacity consistently.  With the new-found synergy of the team, the retrospectives can focus on process improvement and optimizing the operation of the team with the hope of transitioning to the performing phase.
  4. Performing. The performing phase is the goal for all teams but very few actually get to this phase. Not all teams can get to the high level of performance of a Special Ops team where each member knows exactly what their team mates are doing and what they need to do to work as a cohesive unit.  A team in performing phase no longer needs a coach or supervision but can operate autonomously and efficiently.

3.2 Team changes everything

With self-organizing teams, decisions are now made at the team level rather than at the PM or manager levels.  This can be a difficult transition for both the team members and the managers.  Team members are often use to others making the decisions for them and wait for decisions or are reluctant to make decisions.  Managers often feel disempowered when decisions are no longer in their hands because the decisions are made by the team.

The team still has the responsibility to do the traditional activities such as; analysis, design, development testing and deployment.  However, under the new model, the team also takes on the inception responsibilities of: developing user stories, sizing the stories, estimating the stories, creating the product and architectural vision, and developing the release plan.   During construction the team also has to: plan the iteration, do look ahead planning, and have an iteration review (i.e. demo) with the stakeholders.  To keep the team working together they need to have the daily coordination meeting (i.e. Scrum) and to keep the team growing and evolving of course they need to do the retrospective at the end of each iteration.

Thinking, analyzing, planning and building just-in-time is a new way of thinking for a traditional team.  The inception phase in particular can be a difficult adjustment.  Designers and developers are used to doing all the analysis and design upfront.  Doing just enough analysis to be confident of a solution and then writing the user stories to do the work and build the plan is a difficult transition.  Writing user stories so they can get started and learn as they go is a very difficult transition.  Building just the simplest framework and then adding required features and functionality as needed requires practice, training and reinforcement.

There is much more focus on delivering quality because the team is responsible for development, delivery as well as product support.

4.0 The Bottom Line

Chess has only a couple of dozen basic moves and so it is simple to learn the basics.  However, after 3 moves there can be several billion moves available making it so complex that it can take a lifetime to master.  Transforming to Agile is a lot like that.  It looks very simple on the surface but under the covers it can get very complex very quickly.  That’s where the value of a good coach comes in.  You need a coach who brings a lot of experience to the table who will be open and honest with you about what needs to be done and how to get a successful transformation.

What do Reuse Engineers do?

Reuse engineering is an important, and arguably advanced, aspect of the Disciplined Agile framework.  The challenge is that reuse engineering requires significant discipline and organizational maturity to be successful, hence we tend to run into far more talk about reuse than action.  Having said that, many organizations have been very successful with reuse.  The following diagram overviews the internal workflow of a reuse engineering team, capturing key activities (the blue bubbles) that the team performs.  This blog posting explores what reuse engineers do in practice.

Internal workflow of a reuse engineering team

Let’s work through the primary activities performed by reuse engineers:

  1. Guide teams in reuse.  An important activity for reuse engineers is to provide guidance to delivery teams regarding what is available for reuse, how to go about accessing and applying the reusable artifacts, and educating teams in why reuse is important to both the team and to your organization.   Very often a team’s architecture owner will collaborate with the reuse engineering team to bring a reuse engineer into the team at the right time.
  2. Obtain assets.  Reuse engineers will obtain reusable assets from a variety of sources, including from the marketplace and from internal delivery teams.  Based on the various organizational roadmaps, and the needs of the delivery teams that they’re working with, the reuse engineering team will often work with delivery teams to identify and obtain the appropriate assets from the marketplace.  The goal is to find assets that fit the needs of individual teams in a way that aligns with the direction of the organization.  Furthermore, reuse engineers tend to monitor what delivery teams are doing, often by working closely with the organization’s enterprise architecture team and the architecture owners on delivery teams, to help them identify potentially reusable assets.  When a team believes it has built something that is potentially reusable by others, or when the reuse engineers believe they have done so, then the reuse engineers will work with the team to understand and harvest that asset.
  3. Publish reusable assets.  An asset is potentially reusable when it is of high quality, it is appropriately documented, one or more examples exist of how to use it, and it is findable by others.  The publishing process ensures that all of these criteria are true.  The reuse engineers will do the work to publish the asset, refactoring and documenting it as needed and harvesting any usage examples if available (and creating some when not).  After doing so, they will save the asset and its related supporting artifacts into your organization’s reuse repository, announcing the availability of the new asset after doing so. Note that reuse repositories, also called asset managers, have fallen out of favor in the past few years due to the complexity of the available products on the market and the propensity of organizations to use products like Git or Microsoft SharePoint as repositories.
  4. Configure asset for specific use.  Reuse engineers will often work with a team to help them to configure an asset for specific use.  The goal is to make it as easy as possible for others to reuse existing assets, thereby increasing the chance of rate of successful reuse within your organization.
  5. Integrate reusable assets into a solution.  Reuse engineers will often work with delivery teams to integrate a reusable asset into their solution, once again to make it as easy as possible for teams to reuse existing assets.  Interestingly, an important aspect of harvesting an asset for reuse is to help the source team to integrate the improved version of the asset back into their solution.  This helps to increase the likelihood that teams will offer up potential assets for harvesting and publishing.
  6. Evolve reusable assets.  Assets need to evolve over time to reflect changing requirements and implementation technologies.  The implication is that the owners of the assets, often the reuse engineering team, will need to evolve their assets, publish new versions, and deprecate old versions.  This is work that must be funded and supported properly.

If your organization is serious about reuse engineering then they will explicitly fund a reuse engineering team and enable them to work in the manner that we’ve described here.  For more information, you will find the article Reuse Engineering to be of value.