“Fast, Not Furious” – Change Management at the Speed of DevOps

[ericledyard]

“Fast, Not Furious” – Change Management at the Speed of DevOps

It is very common for us to talk about the “Unicorn Companies” vs the “Traditional/Legacy Organizations” when we are discussing DevOps transformations. The “Unicorn Companies” are companies that are born-in-the-cloud, built from the ground up as cloud-first and Agile, and are typically younger upstart organizations that fly below the radar of regulations. Whereas, the “Traditional/Legacy” companies are companies that are well-established and have been around for 30, 50, 100, or even 200+ years and are heavily regulated in their industries. The Unicorns tend to set the bar for “what good looks like” in the DevOps space. They talk about numbers like: “186,000 production changes per day.” And the reaction that most of the traditional organizations have is simply: “How?! That’s impossible.”

When you dig into their reaction, you will see that for traditional companies there is a whole other layer of regulations that the Unicorns never see and are not beholden to. Those systems of safety and controls, while they tend to get in the way of DevOps transformations, are there for one very important reason – to eliminate risk and downtime. We have numbers from both Gartner and Forrester on what the average cost of downtime is for an average company. Gartner estimates it at somewhere around $5600/Minute or over $300,000/hour on average in this blog: Here.

 

Figure 1 - Regulated Companies must bridge the gap between Speed and Safety

In conversations with customers, they tell us that they want to bridge the gap between the speed and agility of DevOps and the Safety and Control of Regulations. The question is not: “Should we adhere to regulations?” It’s “we MUST adhere to regulations and if you can do DevOps also – great. But regulations first.”

This is a challenge for most enterprises as they try to scale Agile and DevOps practices beyond just pockets of innovation to a standard operating procedure. Governance bodies within the organization and operations teams are challenged to accept the rapid pace of innovation and release that Agile teams want to drive. We find this is because the manual processes of Change Management, Release Management, GRC, and Audit are creating bottlenecks to the overall progress of rapid release. I mentioned in a previous blog (citing this Klaus Leopold video) that many companies have DevOps teams who are writing code very quickly and are considering themselves “Agile”, but their release frequency has not improved. This is because they are not looking at the entire value stream to see where improvements need to be made to process to enable flow throughout the entire system. You cannot be successful in your DevOps journey if you only improve “Dev”. The governance requirement can lead to friction between dev and ops teams. 

So, how do you address the “Ops” side of the DevOps transformation, make the change process fast and avoid making devs furious? Well, most IT organizations worth their salt transformed to adopt ITIL to define the standards for their IT Operations. In the DevOps Handbook (Kim, Humble, Debois & Willis) Gene Kim addresses a common misconception that DevOps is Incompatible with ITIL. As you can see below, the solution is not to get rid of ITIL practices – the solution is to AUTOMATE your ITIL practices so they can keep up with the Dev side of the transformation.  Some in DevOps are advocating for moving the governance processes left into the toolchain but this doesn't solve all the problems for large and regulated companies.

 

Figure 2 - DevOps Handbook Myth Debunking around ITIL

The 3 biggest pain points we have found most organizations face from the Ops side of the house are: Change Management, Audit, and Traceability across their DevOps toolchain. Without automating these aspects of the DevOps equation – you cannot increase your release frequency and ability to drive value for the Enterprise. Our direction is that an organization needs to implement technologies that enable Automated Change Management, Push-Button Audit, and provide visibility through analytics across their combined DevOps toolchain. This is how you begin to achieve the value at scale that companies seek by kicking off a DevOps transformation.

The best approach for solving this is to have a data model that allows you to bring data into a centralized location from any DevOps tool and have a highly extensible API structure that allows tools to interact with your centralized platform. This allows for your developers to stay in their development tools and write code, but still capture the data to provide governance behind the scenes so the processes of ITSM can be automated without taking any time or effort from the developers.

For Automated Change Management, you need an engine that can take API calls from development tools like GitHub and Jenkins and create automated change requests for commits, which then go through an automated change approval engine that leverages input triggers, decision trees, and predefined answers to provide a set of business rules that can automatically determine if a change can be approved. It can also add human intervention when needed to ensure change policies are being adhered to and to stop bad changes from making it into production. We have already observed companies going from a total change time of 23+ days down to <5min for 85%+ of changes. That is a huge operational impact and can increase release frequency for the first time in many organizations.

Push-Button Audit is accomplished by gathering all the data throughout the process and capturing the chain of custody that took place throughout the end-to-end lifecycle of a release. From Ideation – (This is where the idea originated. This was the value we were providing. These are the Epics and stories that this release was a part of.) to Development – (These are the commits that were made. This is who made the commits. These are the files that changed.) to Release – (These are the jobs that ran. This is what was in the build. These are the tests that were ran.  These are the test results. These are the automated change policies that were applied.) to Production – (This is when we released to Production.) to Operations – (This was the change that caused an incident. This is how we rolled back.) By capturing all the data from Planning tools, Coding tools, Pipeline Execution tools, and Operations tools – we can build and provide a push-button audit report for an auditor. On average, we find it takes a company 3+ FTE weeks of effort to gather data for an audit. This takes expensive development resources away from writing code to track down audit materials for an auditor. By maintaining a real-time push-button audit trail – it’s as easy as running a report. Even the audit process itself runs more smoothly because the automated population of data reduces the likelihood of data anomalies.

Finally, it is extremely important to understand what is going on with your end-to-end pipeline. Visibility in the past is challenging because no single system has all the data to be able to provide analytics across the disparate DevOps toolchain. In many cases, the tool chain is 10-20 software products being used by teams to deliver code. But none of those tools are integrated to each other and many of the tools do not provide analytics of any sort. We believe the future for DevOps is a platform that can gather data from all the disconnected tools and be able to provide a strong analytics platform that can provide visibility and traceability into what is going on at every stage of the software development lifecycle.

The concepts we are pushing forward in the industry are laying the groundwork for making ITSM and ITIL an integral part of a DevOps strategy. It is resonating with everyone I have spoken to at conferences, analysts, or customers in the field. The simple questions you can ask different actors like: (To the Change Management Team) – “Wouldn’t it be great if you could keep up with all the change requests coming but not lose any control or governance in the process and actually increase the data collection, standardization, and ability to scale change management at the speed of DevOps?” Or, (To the Developers) - “Wouldn’t it be great if you could stay compliant but never have to leave your development tools and all of the data collection and governance happened automatically via API behind the scenes?”

The vision we have for the future is the concept of: “Fast, not Furious.” We want companies to be able to achieve their Agile goals – but also stay compliant with regulations and not cause outages in production. This is the nirvana of DevOps and it is possible if you architect your environment correctly.