A guide to DORA metrics

Annual DevOps surveys provide a wealth of information about the characteristics that high-performing organizations share. The DevOps Research and Assessment (DORA) group sets the industry standards for metrics that track and organize this data to accurately assess an organization’s DevOps capabilities.

From this data, a pattern linking success to the mastery of technical practices has emerged. The DORA group outlines specific metrics to track and work towards in order to get the most out of your product.

In this article, you will learn what the DORA metrics are, how you can calculate them, and why you should implement them within your product team.

What is DORA and what are its findings?

Since 2014, there has been an annual survey of DevOps practitioners that has steadily grown to have had more than 30,000 respondents.

Seeing the wealth of information available from these surveys, the DevOps Research Associates (DORA) team — started by Dr. Nicole Forsgren, Jez Humble, and Gene Kim — has parsed the data, and prepared annual reports.

It found that high-performing organizations tended to have:

• Pace of deployments — The highest-performing organizations deploy hundreds of times more frequently and have lead times thousands of times faster in comparison with low-performing organizations
• Software stability — The highest-performing teams recover from incidents thousands of times faster than the lowest-performing teams
• Security — High-performing organizations spend about half the time addressing security issues in comparison with low-performing organizations

Further analysis of the latest survey data by the DORA team revealed four types of DevOps organizations:

• Starting — The starting cluster features organizations that are mainly in the early stages of development, and as a result, they tend to focus less on reliability because they are still getting feedback on their products and services
• Flowing — The flowing cluster does well across all areas, where teams with this level of performance have high levels of reliability, stability, and throughput. It’s important to point out that only 17 percent of survey respondents achieved this level of performance
• Slowing — The slowing cluster is a reflection of a common reality in organizations that deploy relatively infrequently, while also enjoying reasonably high success rates with those deployments. More than a third of survey respondents are part of this cluster
• Retiring — The retiring cluster is characterized by teams that are working on a product or service that still has business value, and where that product or service is essentially in a maintenance mode

What are the 4 DORA metrics and how can you calculate them?

Now that we’ve looked at general DORA trends, let’s examine the 4 DORA metrics. The first two — deployment frequency and lead time — focus largely on speed (throughput), while the second two — change failure rate and time to restore service — focus on stability:

• Deployment frequency — The relative frequency with which an organization releases software to customers
• Lead time for changes — The elapsed time it takes to get from an initial commit on a feature to that feature reaching customers
• Change failure rate — The relative frequency with which deployments go sideways before getting to customers
• Time to restore service — The amount of time it takes on average to recover from a customer-impacting incident

Deployment frequency

As its name suggests, deployment frequency indicates how often an organization successfully releases software to end users. This metric is an important data point and the DevOps survey data identified three different levels of organizational performance:

• High — On-demand — multiple deployments per day
• Medium — One deployment per week to one per month
• Low — One deployment per month to one every six months

Deployment frequency can vary a great deal from business unit to business unit and even team to team. That being said, the survey data clearly shows that frequent deployments are strongly correlated with high-performing in organizations.

Calculating deployment frequency

You can calculate deployment frequency by dividing the total number of deployments made in a given period of time by the total number of days in that period.

For instance, suppose that you’re focusing on the number of deployments per month and your team deployed code 10 times a month during a 31-day month. In this case, your team’s deployment frequency would be 10 divided by 31, or 0.32 deployments per day.

Lead time for changes

Lead time has been a key metric for decades in organizations that practice lean software development and leverage proven agile values, principles, and practices. Within the context of DORA metrics, lead time refers to the average amount of time that elapses between committing new code and releasing that code into production.

Lead time for changes is a key indicator because the shorter the amount of time that elapses, the greater the organization’s ability to get timely feedback from customers. This often leads to additional improvements to the software based on that feedback.

Shorter feedback loops are a hallmark of lean-agile organizations. The DORA team’s analysis of the DevOps survey data found three levels of lead time for changes:

• High — Between one day and one week
• Medium — Between one week and one month
• Low — Between one month and six months

There are many factors that directly impact lead time for changes. Some of those factors are related to organizational culture and team structure, such as the extent to which software changes need to be approved by separate testing teams and/or decision-makers focusing on compliance considerations such as security and privacy.

Technical factors inform lead time for changes as well, such as the number of software environments that exist, how much test automation is in place, and the maturity of technical practices such as continuous integration (CI) and continuous delivery/continuous deployment (CD).

Calculating lead time for changes

To calculate lead time for changes, you need to pull data for the timestamp associated with code commits. Specifically, you’ll need timestamps for the initial code commit and the code push to production.

Thus, a formula for computing lead time for changes would need to take the median of the initial commit timestamp subtracted from the push to production timestamp.

For a detailed example of how to calculate the lead time for changes, see the DORA lead time for changes.

Change failure rate

The change failure rate provides visibility into how often a deployment to production needs an immediate fix to address a degradation in product performance or an outage.

It’s important for organizations to monitor the change failure rate because the more time they need to spend on addressing problems, the less time they have available to deliver new features to customers. Furthermore, problems in areas such as product reliability and availability negatively impact the organization’s reputation. Analysis of the DevOps survey data shows the following three benchmarks for levels of organizational performance:

• High — 0–15 percent
• Medium — 16–30 percent
• Low — 31–46 percent

Calculating change failure rate

To calculate the change failure rate, it’s necessary to have the following available:

• The total number of production deployments
• The number of remediation-only production deployments (such as hotfix, rollback, patch)
• The number of production deployments that resulted in an incident

Note: When calculating change failure rate, you need to agree on definitions of what each of the terms above means.

To calculate the change failure rate, you start by subtracting the number of remediation-only deployments from production deployments, which gives you the number of failed deployments. Then you divide the number of failed deployments by the total number of production deployments.

Time to restore service

The last of the four DORA metrics is time to restore service. DORA found the following three levels when considering the perspective of general organizational performance:

• High — Less than a day
• Medium — Between a day and a week
• Low — Between a week and a month

Calculating time to restore service

To calculate time to restore service, you’ll need to have a shared understanding of what incidents you’re including as part of your analysis. Once you’ve done that, it’s a reasonably straightforward calculation, where you divide the total incident age (in hours) by the number of incidents.

Let’s say that for the time period you’re looking at, you had five incidents that happened, where two lasted for an hour, two lasted for two hours, and one took three hours to resolve. Thus the calculation would be: (1 + 1 + 2 + 2 + 3) / 5 = 1.8 hours.

Tooling considerations

There’s a constant influx of tools related to DORA metrics. Because of this, rather than suggesting a particular tool or tool suite, let’s take a look at some general guidance. You should consider:

• Integration with existing software delivery and deployment tooling
• Availability of dashboards and similar data visualizations
• Capability to export and share data in formats readily digestible by stakeholders

There are also a couple of free resources to be aware of:

• DevOps quick check
• Four Keys

DevOps Quick Check

The good news is that even in organizations that are just getting started, it’s not hard to get a quick read on how you’re doing. The DevOps Quick Check consists of five multiple-choice questions that provide immediate visibility into how you’re doing, along with possible next steps:

Four Keys

One of the most daunting challenges with DORA metrics comes down to data collection. In addition to various commercial tooling options, there is also an open-source alternative from Google’s Four Keys. Four Keys works through:

• An extract, transform, and load (ETL) pipeline ingests data from a code repository (like GitHub or GitLab), through Google Cloud services, and into Google DataStudio
• Four Keys makes it possible to aggregate and compile the data so it’s visible on dashboards, providing insight into each of the four DORA metrics, which DevOps teams can use to track their progress over time:

Benefits of using DORA metrics

There are numerous advantages to capturing and evaluating data associated with the four DORA metrics. Those advantages include:

• Improved decision-making — One of the biggest challenges is how to deal with the mountain of data that systems, services, and products produce. By narrowing the focus to the data points associated with the DORA metrics, you not only get visibility into your own processes and systems, you also gain insights into areas where changes are likely to make the biggest difference, and prioritize accordingly
• Enhanced ability to deliver value — The better that you do with the DORA metrics, the greater your ability to deliver products that our customers love. Even if you’re in a lower performance tier, DORA metrics show where you’re getting stuck and help get you on a path to more consistent delivery
• Continuous improvement — Teams that work in a lean and agile context thrive when steps to continuous improvement are apparent. You can baseline your performance with DORA metrics and similar measures, and outline areas you can improve

Things to keep in mind when using DORA metrics

When using DORA metrics, especially if you’re just getting started, remember:

• There’s the potential for inconsistency when gathering data, especially in larger organizations. Reaching agreement on definitions of terms and on any limitations in the data is important
• Because the data you need for DORA metrics often comes from multiple tools and applications, any changes to tooling could impact the quality of the data that you have available
• You need to be especially cautious when comparing your own organization’s performance with others. While it’s helpful to see where you fall across the broader landscape of organizations, you need to recognize your own unique context and how it contributes to your current set of capabilities

Conclusion

Because there’s so much data available related to the DORA metrics, seeing how you’re doing in each of the four areas gives you a quick read on your current capabilities. It’s especially important to understand any areas where you’re falling short, and steps you can take to bring yourself closer to your competitors.

Even if you’re already performing at a high level, monitoring your performance against the DORA metrics helps ensure that you continue to surface areas that need improvement.

Featured image source: IconScout