An important aspect of Support that is easily forgotten is the need to build out your infrastructure to enable your support efforts. This may include:
- Creating a support knowledgebase so that your Support Engineers can capture solutions to the problems they solve.
- Provide access to the support knowledgebase to support self-service by end users. This access is often limited for privacy reasons – end users should have access to solutions to common problems but not the details to specific incidents.
- A support environment to simulate problems. In some cases, such as an online trading system perhaps, you don’t want your Support Engineers trying to diagnose end user problems on the live system itself due to potential side effects of doing so.
- Installing communication systems such as chat software and a phone/call in system.
- Automated support systems such as integrated voice response (IVR) and artificial intelligence (AI)/bots
Figure 1. High-level architecture for a Support environment (click on it for larger version).
The Disciplined Agile (DA) framework describes strategies for how an organization’s IT group can support a lean enterprise. An important part of this is to have an effective IT operations strategy, and to do that the people involved need to have what we call a “lean IT operations mindset.” The philosophies behind such a mindset include:
- Run a trustworthy IT ecosystem. At a high level the goal is to “keep the lights on.” At a detailed level anyone responsible for IT operations wants to run an IT ecosystem that is sufficiently secure, resilient, available, performant, usable, and environmentally friendly. Part of running a trustworthy ecosystem is monitoring running services so as to identify and hopefully avoid potential problems before they occur. For some systems, and perhaps for your IT ecosystem as a whole, you may have service level agreements (SLAs) in place with your end users that guarantee a minimum level of trustworthiness.
- Focus on the strategic (long-term) over the tactical (short-term). Anyone responsible for IT operations needs to have a very good understanding between the long-term implications of a decision versus the short-term conveniences. A classic example of this right now is the preference of people building micro-services to use what they believe to be the best technologies for each service. This makes a lot of sense from the narrow viewpoint of that service and it often proves to be incredibly convenient, and fun, for the developers because they often get to work with new technologies. However, from an operational point of view you end up with a mishmash of technologies that must be operated and evolved over time, resulting in a potential maintenance nightmare. Yes, you will still make some short-term decisions but you should do so intelligently. Too great a focus on the long term results in a stagnant IT ecosystem, too great a focus on short-term decisions results in operations teams who spend all their time fighting fires. The long-term technical vision for your organization is developed by your Enterprise Architecture efforts and the long-term business vision comes from your Product Management activities.
- Streamline the overall flow of work. This arguably should be part of everyone’s mindset, but it is particularly important for people doing IT operations work. IT operations has traditionally been a bottleneck in many organizations, often the result of the need to run a trustworthy ecosystem and to focus on long-term considerations, hence the need to focus on streamlining the overall flow of work. BUT, this isn’t just operational work that we need to streamline, but the overall flow of work into, within, and out of IT operations. In this case we need a disciplined approach to DevOps that takes all aspects of the development-operations lifecycle into account, including the support of multiple development lifecycles (not just continuous delivery), the release management process, and the operational aspects of data management. Of course, streamlining the flow of work goes beyond development-operations and is an important goal of your organization’s continuous improvement strategy.
- Help end-users succeed. An important goal of people performing operations activities is to ensure that your end users are successfully using your IT systems. It doesn’t matter how well your systems are built, or how trustworthy they are, if your end users are unable or unwilling to use them effectively. End users are going to need help – you need to be prepared to provide a support function.
- Standardization without stagnation. The more standardized your IT ecosystem is the easier it will be to run, to release new functionality into, and to find and fix problems if they should arise. However, too much standardization can lead to stagnation where it becomes very difficult to evolve your ecosystem. You will need to work very closely with people performing enterprise architecture and product management activities to ensure that you understand the long term vision and are working towards it.
- Regulate releases into production. Most DevOps strategies reflect the viewpoint of a single product team. But what about the viewpoint of your overall IT ecosystem, which may comprise hundreds of products? An interesting question to ask is what is the WIP limit for releases across your overall ecosystem? In other words, what rate of change can your infrastructure, and your stakeholder community, bear? In the Disciplined Agile framework this philosophy is an important driver of the Release Management process blade. Furthermore, some regulatory compliance regimes call out a separation of concerns pertaining to release management – the people building a product are not allowed to release the product into production, someone else must make that decision and do the work (even if “the work” is merely pressing a button to run a script).
- Sufficient documentation. Yes, there will be some documentation maintained about your IT ecosystem. Hopefully this documentation is concise, accurate, and high-level. Common documentation includes an overview(s) of your infrastructure, release procedures (even if fully automated, there’s still some overview documentation and training), and high-level views of critical aspects of your infrastructure including security, data architecture, and network architecture. Organizations that operate in regulated industries will of course need to comply to the documentation requirements of the appropriate regulations. When infrastructure components are discoverable and self-documenting there is a lesser need for external documentation, but there is still a need. Any documentation that you do create should be maintained under configuration management (CM) control.
Future blog postings in this series about IT operations and support will explore topics such as why you need IT operations and support, what activities you perform, and the workflow of doing such.
According to the Data Management Body of Knowledge, data management is “the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets.” In our opinion this is a very good definition, unfortunately the implementation of data management strategies tends to be challenged in practice due to the traditional, documentation-heavy mindset. This mindset tends to result in onerous, bureaucratic strategies that more often than not struggle to support the goals of your organization.
Having said that, data management is still very important to the success of your organization. The Disciplined Agile framework promotes a pragmatic, streamlined approach to data management that fits into the rest of your IT processes – we need to optimize the entire workflow, not sub-optimize our data management strategy. We need to support the overall needs of our organization, producing real value for our stakeholders. Disciplined agile data management does this in an evolutionary and collaborative manner, via concrete data management strategies that provide the right data at the right time to the right people.
There are several reasons why a disciplined agile approach data management is important:
- Data is the lifeblood of your organization. Without data, or more accurately information, you quickly find that you cannot run your business. Having said that, data is only one part of the overall picture. Yes, blood is important but so is your skeleton, your muscles, your organs, and many other body parts. We need to optimize the whole organizational body, not just the “data blood.”
- Data is a corporate asset and needs to be treated as such. Unfortunately the traditional approach to data management has resulted in data with sketchy quality, data that is inconsistent, incomplete, and is often not available in a timely manner. Traditional strategies are too slow moving and heavy-weight to address the needs of modern, lean enterprises. To treat data like a real asset we must adopt concrete agile data quality techniques such as database regression testing to discover quality problems and database refactoring to fix them. We also need to support delivery teams with lightweight agile data models and agile/lean data governance.
- People deserve to have appropriate access to data in a timely manner. People need access to the right data at the right time to make effective decisions. The implication is that your organization must be able to provide the data that an individual should have access to in a streamlined and timely manner.
- Data management must be an enabler of DevOps. As you can see in the following diagram, Data Management is an important part of our overall Disciplined DevOps strategy. A successful DevOps approach requires you to streamline the entire flow between delivery and operations, and part of that effort is to evolve existing production data sources to support new functionality.
In future blog postings we will explore the goal diagram of the Data Management process blade and the associated workflow.
In IT, release management encompasses planning, coordinating, and verifying the deployment of IT solutions into production. Release management requires collaboration by the IT delivery team(s) producing the solutions and the people responsible for your organization’s operational IT infrastructure. In the case of organizations with a “you build it, you run it” DevOps mindset these people may be one in the same, although even in these situations you will often find a group of people responsible for governing the overall release management effort.
There are several reasons why enterprises adopt release management strategies:
- They have a complex operational infrastructure. The greater the complexity of your operational infrastructure the greater the risk that the release of new functionality into production will break something, hence the greater need for release management. Operational infrastructures become complex when there are many technologies in place, when there are different versions or configurations of those technologies, and when solutions are highly coupled to one another. Ideally you should strive to pay down this technical debt.
- There are many delivery teams working in parallel. Your operational infrastructure is a shared environment, or more accurately a collection of shared environments, that your IT delivery teams deploy into. As the number of delivery teams rises, the greater the chance that their release efforts will conflict with one another.
- IT delivery teams need help to release their solutions into production. Your IT delivery teams, particularly new ones, may not have much experience deploying solutions into your operational environment. Your release management team can coach your IT delivery teams in effective release strategies, can guide them in ensuring that their solutions are production ready, and can help in the planning and coordination of their release efforts.
In this continuing series about how the Disciplined Agile Delivery (DAD) framework supports and enhances DevOps, we overview several strategies to help your organization adopt a Disciplined DevOps strategy:
- Invest in your people. In our experience 80 to 90% of your overall effort will be in helping people to learn new skills and ways of thinking and to rethink if not abandon many of the “best practices” of yesteryear. This requires training, education, and coaching over a long period of time – most people will require many months, and sometimes years, to make the transition to this new mindset.
- Create a safe learning environment. Teams must be free to experiment, to try new strategies to discover what works for them in the situation that they face. Very often this will work out well, with a few stumbles along the way, but every so often the experiment will show that the strategy just isn’t right for this team. That should still be considered a successful experiment, learning what doesn’t work is just as valuable as learning what does, and the team should not be worried about recrimination for “failed” experiments.
- Look at the “whole system”. Disciplined DevOps is about more than just continuous delivery (although that is a great start) and in most enterprises it’s about more than just streamlining development and operations. With a Disciplined DevOps mindset we strive to improve flow through the entire ecosystem, including development, operations, support, enterprise architecture, data management, release management, and most importantly to the business itself.
- Improve locally, transform globally. Each team, including your solution delivery teams, your enterprise architecture team, your operations staff, and many others must strive to improve and streamline the way that they work. These local improvement efforts must be supported by a “global” transformation effort that focuses on improving DevOps across your entire ecosystem. Every team will affect other teams, motivating them to make improvements which in turn affects how they work with others. Your IT department is a complex adaptive system where people and teams learn and improve over time in a dynamic, evolutionary manner. If you just focus on local improvements your DevOps effort is likely to devolve into chaos. If you just focus on a company-wide, global transformation it is likely to get bogged down in bureaucracy. An “improve locally, transform globally” approach is a viable middle ground that benefits from these two extremes while avoiding the disadvantages.
- Have a communication plan (and work it). Adopting a Disciplined DevOps strategy within your organization typically requires a lot of (often small) changes. Although it may be clear to you why this is important it isn’t always clear to everyone else. People need to understand why you’re making these changes, what’s in it for them, what the overall change strategy is, how far along the plan you currently are, what changes are coming soon, and so on. You communication plan may include regular newsletters, posters overviewing key concepts, brown bag lunches where people share their experiences, electronic discussion forums, management presentations, and many more. The keys to success are to have a constant drum beat of information, to be as open and honest about what is actually occurring, to provide opportunities to everyone to learn, and to motivate everyone to share their learnings.
- Think long-term. Disciplined DevOps is a journey, not a destination. It takes time for people to adopt a new mindset, months and often years before it is truly ingrained in their way of thinking. This paradigm shift does not occur by management fiat, nor does it occur as the result of a day or two of training (although training is important), nor does it occur because you’ve invested in new tooling. Organizations that successfully make this paradigm shift do so by investing in their people, their process, and their tooling over the long term.
In this posting we briefly explore why both you and your organization should consider adopting a Disciplined DevOps mindset. For yourself as an individual there are several interesting benefits. First, you become more productive as an IT professional, increasing your chance at promotion and making you more attractive in the marketplace. Second, you are in a position where you can focus on interesting, value-added work, which should lead to greater job satisfaction for you. Third, much of the dysfunctional politics exhibited in traditional IT organizations is effectively squeezed out as you move to a Disciplined DevOps mindset, making your work environment a more enjoyable place to be.
There are many reasons why your organization should consider adopting a Disciplined DevOps mindset. The following table summarizes the potential benefits and how they are achieved:
|Decreased time to market
- Shorter Transition efforts from automation
- Smaller “chunks” of work can be implemented faster
|Decreased cost to deploy
- Automated regression testing
- Automated deployment
- Streamlined release management
|Improved mean time between deployments
- Practices such as Continuous Integration and Continuous Delivery enable teams to deploy more often
- Decreased cost to deploy enables teams to deploy more often
- Adoption of agile testing and quality techniques such as automated regression testing, refactoring, independent testing, and many others
- Agile and lean strategies are applied to enterprise architecture, enabling a more holistic view of the organization which in turn promotes greater reuse and reduction/avoidance of technical debt
- Agile and lean strategies are applied to data management, improving overall data quality across your organization
|Improved market competitiveness
- Real-time insight from Development Intelligence strategies
- Real-time insight from Operational Intelligence strategies
- Shorter feedback cycles provided by decreased time to market enable teams to easily run experiments to discover what their stakeholders actually want
As always, we welcome any feedback that you may have.
The Disciplined Agile Delivery (DAD) framework explicitly includes architecture-related activities, the role of Architecture Owner, and promotes the philosophy of enterprise awareness. Our experience is that agile enterprise architecture proves to be a key enabler for organizations in the process of adopting a Disciplined DevOps mindset.
In addition to general DevOps strategies , there are several enterprise architecture activities that support DevOps:
- Reuse mindset. An important thing that your enterprise architecture efforts will accomplish is the promotion of a reuse mindset within IT, and throughout your organization in general. Delivery teams with a reuse mindset strive to leverage existing data sources, services, components, frameworks, templates, and many other assets. This reuse mindset is enabled through education, coaching and mentoring by your enterprise architects (who are ideally active members of IT delivery teams in the role of Architecture Owner). It is also enabled by technical roadmaps that indicate the technologies that IT delivery teams should, and shouldn’t, be working with. And of course, having high-quality assets that are easy to discover, to understand, and to apply in the course of providing real value to your stakeholders enables reuse.
- Technical-debt mindset. Your enterprise architecture effort should promote strategies that motivate delivery teams to pay down technical debt when they find it and more important do what they can to avoid it in the first place. Many technical debt strategies are embedded right in the DAD framework, but without a technical-debt mindset this often comes to naught. Enterprise architects, often acting as Architecture Owners on delivery teams, should coach and mentor developers around the issues associated with technical debt. Similarly they should help to educate the senior managers and stakeholders whom they collaborate with in technical debt as well. It requires investment to avoid and remove technical debt, and IT investment decisions are typically in the hands of these people.
- Development guidelines. An important aspect of enterprise architecture is the development of guidelines for addressing common concerns across IT delivery teams. Your organization may develop security guidelines, connectivity guidelines, coding standards, and many others. By following common development guidelines your IT delivery teams produce more consistent solutions which in turn makes them easier to operate and support once in production, thereby supporting your DevOps strategy. A potential drawback of common development guidelines is that developers may feel constrained by them. To counteract this problem the guidelines should be developed and evolved in a collaborative manner with the delivery teams, not imposed from above.
- Technical roadmaps. Your enterprise architecture efforts include the definition, support, and evolution of technical roadmaps that guide the efforts of the rest of the organization (business roadmaps, also important, are the purview of Product Management). This in turn supports the creation of a common and consistent technical infrastructure within your production environments, enabling common DevOps practices such as continuous deployment, automated end-to-end regression testing, and operational monitoring that we discussed in previous blog postings.
An important aspect of your technical roadmap is to capture both the existing IT infrastructure and the future vision for that infrastructure. Your IT infrastructure potentially includes your network, software services, servers, middleware, and data sources to name a few elements. As you can see in the following diagram, when developing your technical infrastructure vision there are two issues to consider:
- Ownership. Does your organization own and operate its own infrastructure or does it outsource some or all of it to external experts? Outsourcing options include traditional strategies such as having another organization (such as HP or IBM) run your data centers to using cloud-technologies hosted by external organizations (such as Amazon or Google). The advantage of owning your own infrastructure is the greater level of control that it provides you, something that is critical when you must guarantee the security and integrity of your IT solutions. Outsourcing potentially offers greater flexibility in managing your IT infrastructure and cost savings from economies of scale. However, outsourcing requires more sophisticated governance and in the case of traditional strategies is a potential bottleneck when the outsourcer cannot respond in a timely manner to your requests.
- Virtualization. Are the elements of your IT infrastructure built to meet the needs of specific solutions or are they softwarized to provide malleability and ease of evolution? With softwarization, also known as software-defined infrastructure (SDI), the elements of your IT infrastructure are fully virtualized. Softwarization includes IT infrastructure models such as a software defined data center (SDDC), software defined storage (SDS), and software defined network (SDN). Softwarization is typically implemented using cloud-based technologies on either side of your firewall. Greater virtualization offers to increase flexibility and programmability of your IT infrastructure, and thereby enabling you to optimize resources. However, the greater flexibility of virtualization can increase the complexity of your testing efforts and make operational incident simulation more difficult.
In the Disciplined Agile Delivery (DAD) framework data management is a Run (operational) activity that focuses on the execution of data-oriented architectures, policies, and processes. Note that the long-term planning efforts around data-oriented aspects of your organization are part of your Enterprise Architecture efforts. Similarly, development of the data-oriented aspects of your organizational eco-system is addressed by solution delivery teams.
Because data management is an important aspect of your Run endeavors it will be affected by your Disciplined DevOps strategy. Our experience is that in addition to the general DevOps strategies describe previously, there are several data management strategies that support DevOps:
- Data and information guidelines. A straightforward way to promote greater consistency in the development and application of data and information sources is to have common guidance that teams will adopt and then follow. This guidance, including both standard policies and guidelines, will need to be defined, supported, and evolved over time in a collaborative and open manner.
- Quality data sources. Your production data sources, including files, databases, and data feeds, should be high quality assets that are easy to work with. When it comes to data sources of record it is particularly important for them to be of high quality so that they are easy to work with and evolve. Unfortunately this is often little more than fanciful thinking in many organizations. With a Disciplined DevOps mindset teams realize that they should be very careful about increasing the technical debt within their data sources, and more importantly invest in the effort to pay down any technical debt that they find.
- IT intelligence. IT intelligence is the creation, support, evolution, and operation of data warehouse (DW)/business intelligence (BI) solutions that support the management and governance of your IT efforts. From a Disciplined DevOps perspective this there are two important aspects of IT intelligence: development intelligence that provides insight into how delivery teams are working and operational intelligence that provides insight into what occurs in production. The automated team dashboards provided by many development platforms are a simple form of development intelligence, a more sophisticated (and useful) strategy is to combine information from various development tools to display it in an integrated dashboard for the team, and more sophisticated yet is to roll up/combine information from different delivery teams into a portfolio management dashboard. Similarly your operations team may have individual dashboards for each solution, they may combine information being generated by individual solutions into an integrated dashboard, and better yet share that information for an IT portfolio management view.
In the next blog posting in this series we will explore DevOps strategies pertaining to enterprise architecture.
In addition to the general release management strategies described previously, the general DevOps strategies, and the construction-focused DevOps strategies (including continuous deployment) there are several other release management strategies that support DevOps:
- Integrated deployment planning. From the point of view of development teams, deployment planning has always required interaction with an organization’s operations and release management staff; in some cases, via liaison specialists within operations often called release engineers. When you adopt a Disciplined DevOps mindset, you quickly realize the need to take a cross-team approach to deployment planning due to the need for operations staff to work with all of your development teams. This isn’t news to operations staff, but it can be a surprise to development teams accustomed to working in their own, siloed environments (luckily this strategy is built into DAD’s Move Closer to a Deployable Release process goal). If your team is not doing this already, you will need to start vying for release slots in the overall organizational deployment schedule.
- Standard development and testing environments based on production. Development teams know that the greater the consistency between their development, testing, and production environments the easier it is to test and deploy. In multi-team environments the implication is that this will result in defacto standardization of many aspects of your environments. Developers may choose different development tools, but aspects of the infrastructure such as operating systems, application servers, middleware, databases and so on will become consistent over time to streamline the overall release process.
- Release service streams. A key tenant of the DAD framework is that every team is unique, and an implication of that is that some teams will need more help than others. Teams will produce different levels of quality, they will have different amounts of automation, they will have different release cadences, and so on. As a result your release management strategy needs to be flexible enough to address these different situations. One way to do so is to offer different server streams, or service levels as it were, to solution delivery teams. For example, you may have a basic release management service stream where release management engineers actively help delivery teams to deploy their solutions into production and even help them to start automating some of their processes. At the other end of the spectrum you may have a continuous delivery service stream for delivery teams that have fully automated their testing and deployment processes and that can be trusted to successfully deploy on their own. And of course you could have several other service streams between those two extremes. The advantage of this approach is that it is very flexible albeit at the cost of slightly greater scheduling complexity.
- Release blackout periods. Some organizations have periods of time where they choose not to release new functionality into production unless it is absolutely critical. These blackout periods typically occur during high-volumes of business transactions. For example, many North American retail companies will have blackout periods between mid-November and early January for the holiday sales seasons. Many organizations will have blackout periods near the end of their fiscal years to enable them to focus on the process required to close out the year.
- Shared practices. Although this is really a process improvement issue, it’s worthwhile to point out that whoever is involved with release management should actively strive to share effective practices between teams. Sharing learnings across teams is an important aspect of enterprise awareness.
In the next blog posting in this series we will explore data management strategies that support a DevOps mindset.
In this blog posting we describe four general release management strategies that support DevOps. These strategies, from least effective to most effective, are:
- Release windows (slow cadence). A release window is a period of time during which one or more teams may release into production. A release slot is subset within that release window (and may be the entire window) during which a team may deploy their solution into production. For example, your organization may have a policy that production releases occur between 1am and 5am on Saturday evenings (the release window) and that up to four releases may occur during that window (the release slots). In lean terms, a release slot is effectively a Kanban card that allows a team to deploy. Release windows tend to align with periods where system usage is lower, although in the modern world of global 24/7 operations these periods have all but disappeared. With a slow cadence approach to this strategy the release windows occur far apart, as seldom as once a week or even longer. The advantages of this approach are that it provides a consistent release cadence to business stakeholders and it provides consistent release date targets for delivery teams. The primary disadvantage with slow cadence release windows is that they become bottlenecks for release management processes that need to support multiple teams. There are only so many release slots available during each window and this number can be easily exceeded, forcing teams to aim for a future release window. This problem becomes exacerbated when teams start to move to a continuous delivery strategy.
- Release train. The idea with a release train is that every team involved with that “train” has the same release cadence – for example this train releases once a quarter, or once a month, or even once a week. This strategy is commonly used in large programs, or teams of teams, where the individual teams are each working on part of a larger whole. Having the common drumbeat of a release train provides a consistent release schedule for stakeholders and serves as a rallying point for development teams. The train metaphor works quite well in practice. If your team misses the release date, if you miss the train, then the train goes on without you and you need to wait for space on the next on. Dependencies are also respected. For example, if several components need to ship together they must all go on the same train (similar to a family taking a trip together). The primary disadvantage is that development teams are constrained to a common release schedule, making it difficult to support lean or continuous delivery strategies. A potential disadvantage is that release trains may also suffer from the bottleneck problems of slow cadence release windows.
- Release windows (quick cadence). To support continuous deployment, particularly across many delivery teams, you will need a large number of release slots. The implication is that you will also likely need more release windows more often. The advantage of quick cadence release windows is that they are less likely to suffer from the bottleneck challenges associated with slow cadence release windows and release trains.
- Continuous release availability. With this approach delivery teams are allowed to release their solutions into production whenever they need to. In many ways this is simply an extension of the release window strategy to be 24/7. This is the only strategy that truly supports continuous delivery. To make it work a host of DevOps practices are required, such as fully automated deployment, fully automated regression testing, feature toggles, self-recovering components, and many others are required.
Our experience is that most enterprises today employ a slow cadence release window approach although are starting to evolve into the quick cadence version of this strategy. This is usually motivated by the adoption of agile techniques by solution delivery teams and more often than not by continuous delivery practices. We also see large programs take a release train approach, a strategy pioneered in the 1990s by large software companies such as Microsoft and Rational that sold software suites comprised of many products that needed to be shipped together. In recent years the OpenUP and SAFe frameworks have popularized the release train strategy. The strategy of continuous release availability is commonly used in advanced DevOps organizations such as Etsy and Amazon.
An important point to be made is that there are several options available to you, each of which has its advantages and disadvantages. No single approach is perfect, and no single approach works in all situations. You not only need to have choices, it’s good to have choices.
The next blog posting in this series continues with more release management DevOps strategies.