We often hear that agile software development is fine when you face a simple problem, but that agile isn’t sufficient for more difficult problems. Of course this falsehood is promoted by people who have little or no agile experience, or who have been involved with a failed “agile adoption” (usually these teams adopted ad hoc strategies thinking they were agile). Anyway, we decided to look into whether agile teams are taking on hard domain problems in practice.
The following diagram summarizes the responses to our question around agile teams and compliance from our 2016 Agility at Scale study. As you can see, 40% of respondents indicated that their agile team faced either complex or very complex problems, and that a further 38% faced medium complexity. Interestingly, only one in eight respondents said that their team faced a straight forward problem.
The bottom line is that agile strategies, and in particular disciplined agile strategies, are in fact applicable for taking on complex problems. More importantly, this is happening in practice around the world on a regular basis.
We often hear that agile is great for simple situations but as soon as you face compliancy issues that it doesn’t work. Is it possible to be agile when you face regulatory compliance, such as PCI and FDA compliancy? Is it possible to be agile when you face organizational compliance, such as working in a CMMI regime? Important questions that we decided to look into.
The following diagram summarizes the responses to our question around agile teams and compliance from our 2016 Agility at Scale study. As you can see, 62% of respondents indicated that their agile team faced some form of regulatory compliance, 20% some form of organizational compliance, and 15% said both. In fact, two-thirds of agile teams operate under one or more compliancy requirements.
For further reading about compliancy, please read our detailed blog posting Agile and Regulatory Compliance.
We are often told that agile teams should be whole, that they should have sufficient people, funding, and skills to fulfill their mission. The idea is that this reduces the dependencies that your team has on others, enabling them to make decisions and to collaborate more effectively. But, is this actually happening in practice? Are agile teams truly whole, or do they still need to collaborate with other teams (hopefully productively) to get the job done? Being strong believers in empiricism over rhetoric we decided to look into this issue.
In November of 2016 we ran the 2016 Agility at Scale survey. It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on. The following infographic summarizes the findings from the question that explored whether agile delivery teams need to work with external teams or groups to get their work done – in other words, are agile teams truly whole or do they rely on others? As you can see, 96% of respondents indicated that in practice their team had to work with one or more other teams, leading to the conclusion that very few agile teams appear to be truly whole.
One of the fundamental principles underlying the Disciplined Agile framework is that disciplined agilists should be enterprise aware – they should recognize that they need to collaborate with others outside of their team, that they should work towards a common organizational vision, and that they should strive to do what is best for the organization and not just what is convenient for them. Given that agile teams are collaborating with others in practice, it is clear that this philosophy of being enterprise aware is important.
The following diagram presents the results from the survey question in greater detail. You can obtain the source data, a copy of the original questions, and a slide deck key diagrams at the 2016 Agility at Scale survey page.
There are several strategies that you can choose to employ with vertically slicing the requirements for a DW/BI solution. These strategies are described in the following table. There are example stories for each strategy as well as some advice for when to apply each strategy.
Table 1. Vertical slicing strategies for a DW/BI solution.
||When to Do This
|One new data element from a single data source
- As a Professor I would like to know the names of my students so that I know who should be there
- As a Student I would like to know what courses are taught at the university
|Very early days when you are still building out fundamental infrastructure components. Very common for the first iteration or two of Construction. These slices still add real business value, albeit minimal.
|One new data element from several sources
- As a Professor I would like the student list for a seminar that I teach
- As a Student I would like to know what seminars are being taught this semester
|Early days during Construction when you are still building out the infrastructure. These slices add some business value, often fleshing a DW data element to include the full range of data values for it.
|A change to an existing report
- As a Professor I would like to know the standard deviation of marks within a seminar that I teach
- As a Student I would like to know how many spots are still available in a seminar
|Evolution of existing functionality to support new decision making
|A new report
- As a Professor I would like to know the distribution curve of student marks in a seminar that I teach so I may adjust accordingly
- As a Registrar I would like to know what Seminars are close to being full
|Several iterations into Construction when the DW/BI solution has been built up sufficiently.
|A new reporting view
- As a Registrar I would like to know what the prerequisites are for a seminar so that I can advise students
- As a Professor I would like to know the current course load of each student within a seminar that I teach
|Several iterations into Construction when the DW/BI solution has been built up sufficiently.
|A new DW/DM table
- As a Chancellor I would like to track the revenues generated from parking pay meters to identify potential profits to divert to supporting students
- As a Professor I would like to recommend suggested readings to help people prepare before taking a seminar
|Several iterations into Construction when the DW/BI solution has been built up sufficiently.
There are several interesting things about the stories in the table:
- They are written from the point of view of your stakeholders. They aren’t a technical specification. For example, the first story describes how professors want a list of student names but it isn’t saying from what data source(s), what the element names are, … These are design issues, not requirement issues.
- They always provide business value. The first story appears to be the beginnings of an attendee list for a seminar. Having something as simple as a list of names does in fact provide a bit of value to professors.
- Sometimes that business value isn’t (yet) sufficient. It may take several iterations to implement something that your stakeholders want delivered into production, particularly at first. For example, although a list of student names is the beginnings of a class list it might not be enough functionality to justify putting it into production. Perhaps professors also need to know the program that the student is enrolled in, their current year of study, and basic information about the seminar such as the course name, time, and location of it. The decision as to whether the functionality is sufficient to ship is in the hands of your stakeholder (this is one of the reasons why you want to demo your work on a regular basis).
I’ve written a detailed explanation of vertical slicing for a DW/BI solution, and of course there is a wealth of information about agile database techniques in general for those of you interested in greater detail. You may also find our one-day Disciplined Agile Data Warehousing/Business Intelligence Workshop to be a valuable learning experience too.
Many people, particularly those new to agile, will tell you that agile teams must be small and co-located. That is certainly a smart way of organizing a team, but is isn’t required. In fact agile teams are more likely to be geographically distributed in some way than they are to be co-located. In practice, not theory.
In November of 2016 we ran the 2016 Agility at Scale survey. It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on. The following graph summarizes the responses around geographic distribution.
The survey found that less than one-third of agile teams are near-located, where all of the IT members are either co-located or at least in a shared open space. Previous studies have found that this number drops to one-in-ten teams being near located when you also include primary stakeholders.
Don’t let anyone tell you that you can’t do agile with a geographically distributed team because others are clearly doing so in practice. Yes, geographically distributed agile is different than near-located agile, which is one of the reasons why you need to take a pragmatic, context-sensitive approach to agile solution delivery. The Disciplined Agile framework provides the foundation from which to scale your approach to solution delivery to address a range of scaling factors, including geographic distribution. In fact, you may find our article around geographically distributed agile teams to be an interesting read.
The contrite answer is that they’re as large as they need to be, and the contrite agile answer is that they’re as small as they can be. Now that we’ve gotten the contrite answers out of the way, how large are agile teams in practice?
In November of 2016 we ran the 2016 Agility at Scale survey. It was targeted at people who were currently working on agile teams, or who had recently worked on agile teams, and we asked them straightforward questions around the size of the team, how distributed it was, what complexities they faced, an so on. The following graph summarizes the responses around team size.
This year’s survey found that roughly half (48%) of agile teams are more than 10 people in size and one-quarter are more than 20 people in size. These findings are similar to what we’ve found in the past with both the 2012 Agility at Scale survey and the 2009 Agility at Scale survey.
In short, don’t let anyone tell you that you can’t do agile with a large team because others are clearly doing so in practice. Yes, large team agile is different than small team agile, which is one of the reasons why you need to take a pragmatic, context-sensitive approach to agile solution delivery. The Disciplined Agile framework provides the foundation from which to scale your approach to solution delivery to address a range of scaling factors, including team size. In fact, you may find our article around large agile teams to be of interest.
We have recently published a new Inception process goal, Develop Initial Test Strategy. The potential need for this goal was identified a little over a year ago by an organization that we were actively working with and since then we have worked with several other organizations where it was also clear that this process goal was needed.
The basic idea is that during the Inception phase the team should consider developing an initial test strategy so as to identify how they will approach verification and validation. This includes thinking about how you intend to staff your team, organize your team, capture your plan, approach testing, approach development/programming, choose a platform for test environment(s) platform, choose a platform equivalency strategy, test non-functional requirements, automate test suites, obtain test data, automate builds, report defects, and govern your quality efforts. The goal diagram is depicted in Figure 1 and the update goals overview for Disciplined Agile Delivery (DAD) is shown in Figure 2.
Figure 1. The goal diagram for Develop Initial Test Strategy.
Figure 2. The process goals diagram for DAD.
We recently updated the Secure Funding goal diagram, shown below. The change was the addition of the Choose Funding Scope decision point, to recognize that the type of team you’re funding will affect how you fund said team. We have also published details about each decision point and its associated options, including the tradeoffs of each, at the Secure Funding overview page. This is information that we previously had only published in the first Disciplined Agile Delivery book but we’ve decided to start publishing updated versions of this information to the web to make it more accessible to you.
Reuse engineering is an important, and arguably advanced, aspect of the Disciplined Agile framework. The challenge is that reuse engineering requires significant discipline and organizational maturity to be successful, hence we tend to run into far more talk about reuse than action. Having said that, many organizations have been very successful with reuse. The following diagram overviews the internal workflow of a reuse engineering team, capturing key activities (the blue bubbles) that the team performs. This blog posting explores what reuse engineers do in practice.
Let’s work through the primary activities performed by reuse engineers:
- Guide teams in reuse. An important activity for reuse engineers is to provide guidance to delivery teams regarding what is available for reuse, how to go about accessing and applying the reusable artifacts, and educating teams in why reuse is important to both the team and to your organization. Very often a team’s architecture owner will collaborate with the reuse engineering team to bring a reuse engineer into the team at the right time.
- Obtain assets. Reuse engineers will obtain reusable assets from a variety of sources, including from the marketplace and from internal delivery teams. Based on the various organizational roadmaps, and the needs of the delivery teams that they’re working with, the reuse engineering team will often work with delivery teams to identify and obtain the appropriate assets from the marketplace. The goal is to find assets that fit the needs of individual teams in a way that aligns with the direction of the organization. Furthermore, reuse engineers tend to monitor what delivery teams are doing, often by working closely with the organization’s enterprise architecture team and the architecture owners on delivery teams, to help them identify potentially reusable assets. When a team believes it has built something that is potentially reusable by others, or when the reuse engineers believe they have done so, then the reuse engineers will work with the team to understand and harvest that asset.
- Publish reusable assets. An asset is potentially reusable when it is of high quality, it is appropriately documented, one or more examples exist of how to use it, and it is findable by others. The publishing process ensures that all of these criteria are true. The reuse engineers will do the work to publish the asset, refactoring and documenting it as needed and harvesting any usage examples if available (and creating some when not). After doing so, they will save the asset and its related supporting artifacts into your organization’s reuse repository, announcing the availability of the new asset after doing so. Note that reuse repositories, also called asset managers, have fallen out of favor in the past few years due to the complexity of the available products on the market and the propensity of organizations to use products like Git or Microsoft SharePoint as repositories.
- Configure asset for specific use. Reuse engineers will often work with a team to help them to configure an asset for specific use. The goal is to make it as easy as possible for others to reuse existing assets, thereby increasing the chance of rate of successful reuse within your organization.
- Integrate reusable assets into a solution. Reuse engineers will often work with delivery teams to integrate a reusable asset into their solution, once again to make it as easy as possible for teams to reuse existing assets. Interestingly, an important aspect of harvesting an asset for reuse is to help the source team to integrate the improved version of the asset back into their solution. This helps to increase the likelihood that teams will offer up potential assets for harvesting and publishing.
- Evolve reusable assets. Assets need to evolve over time to reflect changing requirements and implementation technologies. The implication is that the owners of the assets, often the reuse engineering team, will need to evolve their assets, publish new versions, and deprecate old versions. This is work that must be funded and supported properly.
If your organization is serious about reuse engineering then they will explicitly fund a reuse engineering team and enable them to work in the manner that we’ve described here. For more information, you will find the article Reuse Engineering to be of value.
IT Portfolio Management addresses how an IT organization goes about identifying, prioritizing, organizing, and governing their various IT endeavors. Disciplined Agile Portfolio Management seeks to do this in a lightweight and streamlined manner that maximizes the creation of business value in a long-term sustainable manner. IT endeavors typically include solution delivery initiatives/projects, stable product development teams, business experiments (along the lines of a lean startup strategy), and the operation of existing IT-based solutions.
Being agile, having an agile mindset, is foundational to working in an agile manner. The Disciplined Agile Manifesto and the principles of lean software development provide an important start at this mindset. In this blog we explore similar agile philosophies that are specific to successful portfolio management. These philosophies are:
- Keep it simple. Keep your portfolio management activities as streamlined and lightweight as can be. Your goal should be to focus on making good decisions and providing guidance to people, not on maintaining extensive documentation or reviewing documentation. You still may choose to maintain artifacts such as a portfolio roadmap and portfolio budget, but those too should be as lightweight and concise as possible.
- Focus on value over cost. Shifting your mindset from “what is this going to cost?” to “what value will this generate?” is critical to your success because it helps you to focus making better IT investments. You want to invest in IT endeavors that enhance your organization’s ability to produce value for your customers. This in turn provides the profits required for further investment.
- Reduce the cost of delay. One of the strategies for maximizing stakeholder value is to invest in developing functionality that will provide value to the organization soonest. For example, if you delay developing functionality that will generate annual revenue of $10 million for six months you have an effective cost of delay of $5 million because you missed out on half a year of revenue. Disciplined agile portfolio managers consider the cost of developing a solution, the cost of delay that results from putting off said development, and the revenue generated (or cost savings) when calculating the overall value of a solution.
- Invest in streamlining the creation of value. Not only do we want to produce value for our customers, we also want to be good at doing so. The implication is that we sometimes need to invest in ourselves through process or environment improvements.
- Prefer stable teams over project teams. Although portfolio management is often believed to be the oversight of project teams, it really is more about the coordination and oversight of teams in general. In Disciplined Agile we recognize that an initiative is seldom finished at the end of a “project”. There are usually subsequent changes required over time requiring future releases of the solution. The agile community has discovered that long-lived stable teams, whose membership evolves over time, have significant advantages over short-lived project teams. A significant productivity improvement occurs when IT organizations shift away from the project mindset of bringing people to the work and instead decide to bring work to the people (the stable teams).
- Align teams to value streams. These stable teams should be aligned long-term to value streams or lines of business (LOBs). High performing agile teams reliably deliver value frequently to their business stakeholders. As a result the business learns to trust the teams aligned to their areas. This positive feedback cycle maximizes the effectiveness of your agile teams. Additionally, over time they learn more about the business adding to their effectiveness.
- Enable diversity. Every person, every team, and every organization is unique. Every team faces a unique situation that evolves over time. The implication is that teams must be allowed to organize themselves and to tailor their process to meet the context of the situation that they face. This is why the Disciplined Agile framework focuses on providing, and comparing and contrasting, a wealth of process choices. The implication for portfolio managers is that they need to be flexible in their approach. They will work differently with each team because each team works differently, yet they must still provide good guidance to these teams and monitor the effectiveness of each team appropriately.
- Trust but verify. Effective governance is based on enabling and then trusting your teams to do the right thing. However, effective teams monitor themselves through the use of automated dashboard technology and close collaboration, and the metrics that teams collect should be made visible to senior management and other stakeholders so that they may monitor what is happening.
- Govern by risk not by artifacts. Traditional governance often focuses on the review of common artifacts such as requirement documents, architecture models, and test results. Because it is relatively easy for teams to create the documentation that you want to see, in practice there is very little governance value in reviewing these artifacts. Agile governance instead focuses on addressing common risks such as ensuring there is an agree to vision for what the team should accomplish, that the architectural strategy has been proven to be viable early in the lifecycle, and that the team has produced sufficient business value for their stakeholders.
- Rolling wave over annual planning. Annual planning often begins in earnest mid-year which means that prioritized initiatives may not be actually delivered for up to 18 months in the following year. This is not business agility. Make your planning a continuous “rolling wave” activity year round with more detail devoted to planning initiatives no longer than 6 months out. Initiatives planned beyond 6 months should be described at a very high level.
- Prefer small initiatives over large initiatives. It is a proven fact that the larger the initiative the greater the chance of failure. Smaller initiatives are easier to plan and are lower risk to execute. Keep your initiatives small to allow for more frequent delivery of value with less investment in work in progress (WIP).
- Invest in quality. Ensuring that IT delivery teams produce new business value for your organization is clearly important. But so is ensuring that you will still be able to continue doing so a few years from now. To ensure the long-term sustainability of your IT teams you must allow them to make investments in quality. These investments include building high quality assets in the first place but also fixing low quality assets, what agilists refer to as paying down technical debt, as well.
- Optimize the whole. Disciplined agilists are enterprise aware. We choose to optimize the whole instead of locally optimizing a single activity. The implication is that you cannot consider portfolio management on its own but instead must consider it in the context of the other parts of your organization that it affects. A strategy that may make things easier to manage your portfolio, such as having a single way to fund IT delivery teams, may make it easier for your portfolio management efforts but could inflict undue costs and bureaucracy on the teams that you’re funding. For example, does it really make sense for a team asking for $50,000 in funding to go through the same level of rigor as a team asking for $5,000,000? Likely not. Context counts.
Our experience is that the philosophies describe above enable portfolio managers to be more effective in practice. We hope you have found this blog of value and we welcome your feedback.