The new DORA report is out since November 20th 2024, and it is a great source of information if you want to understand how to help your software team be great and deliver awesome products.
If you have no idea what I’m talking about, here’s a quick intro.
DORA which means DevOps Research and Assessment, is an annual research conducted to identify what makes high-performing technology-driven teams and organizations succeed. They have been doing it for over a decade.
The reason why I want to talk about it here, is because they touch on some Agile concepts, including the implementation of Agile in general. In addition, if you’ve seen some of my more recent videos where I talk about technical knowledge that any Agile leader should have, this report touches on those areas too.
I’ve read the whole report thoroughly. All 120 pages of it 😵 Yes, including the methodology section where they describe how they collected the data and analyzed it. The reason for that is that I want to give you as much context as possible.
You can easily jump around different chapters of this video if you want to skip certain parts.
So let’s jump in! First things first…
Who and what is part of the report?
DORA generally sends out a survey to professionals working in technical and adjacent roles. This year, nearly 3000 people have provided their answers. That’s a lot, but also not as many considering how huge the tech industry is.
They based who should participate off of the Stack Overflow Developer Survey in 2023 where over 90000 people participated. They used this as a census of the developer world.
As they said in the report:
The majority of people identified that they work in:
- Technology, that’s 35.7%.
- Then Financial Services which is 15.6% – not surprising since banks really like to implement agile these days, and then the numbers go down.
- In third place is Retail/Consumer/E-commerce with about 9.5%.
The respondents mostly represented bigger organizations as you can see on this graph:
Interesting statistic here, in my opinion, was that people identified to have between 10 and 25 years of working experience. So we’re looking probably at an older demographic, than people fresh out of college, or going through their first or second job. Because I have just over 15 years of experience, and I started to work fairly early in my life.
And then, when it comes to people’s roles: Developers represent 29% of the respondents, Managers – 23%; Senior executives – 9%, and Analytic roles about 5%.
The report kind of represents all countries, but the respondents are primarily in North America, or Europe, including UK. And India, of course, represents a big chunk. Surprisingly, Japan also had a high representation in the report.
How the data was collected and analyzed
Now in terms of how they actually collected and analyzed the data, real quick. Well firstly, they promoted the survey publicly and encouraged people to participate.
They also had to split their questions and send people to different pathways of questions to avoid making the survey too long. What this means is that out of the 3000 people, not everyone responded to the same questions, so the data is based even on a smaller number of responses.
They had three pathways, so I assume they sent about the same number of people through each.
The reason why I explain all this to you is to basically give you a reason to take everything with a grain of salt. There are a lot of elements here about who participated and how the data was collected, that show that this is a bit of an opinion piece. And it can definitely help us understand a lot about how people work and what makes a team successful, but it’s still just individual perception that can be quite biased.
With this out of the way, let’s review the key findings.
What DORA evaluates
The report doesn’t present us with all of the details exactly, but instead provides an overall summary. I didn’t just read the abstract of the research, but all of it that was available.
The purpose of DORA in simple terms is to understand what factors contribute to the success of teams and organizations.
Every year DORA changes some of the key elements they focus on. This year they looked into the following accomplishments and outcomes when it comes to high-performing teams and organizations:
- Reducing burnout. That’s I think pretty self-explanatory.
- Flow or how much focus developers are able to achieve when performing tasks.
- Job satisfaction measured based on employees feelings about their job.
- Organization performance in relation to profitability, market share, total customers and customer satisfaction, and other business metrics.
- Product performance when it comes to usability, functionality, value, availability, security.
- Productivity measured in the feeling of being effective and efficient in their work.
- Team performance when it comes to collaboration and effective teamwork.
Basically, they were looking at how various factors impacted these outcomes – positively or negatively, and what practices high-performing organizations use, where high-performing organizations usually have all of the outcomes mentioned at their best level.
The report is split into several sections and this is how I’m gonna split this video going forward.
The report had a lot, and I mean, A LOT of space dedicated to AI. Honestly, I couldn't care less about this because none of their findings were particularly interesting or useful. Everyone is obsessed with AI these days... 🙄, which I'm not necessarily interested in. If you want, you can read the report for more details, but I'll be skipping it in my video.
Software delivery performance
DORA uses four keys as they call them to measure how well the software delivery process works. Heads up to those of you who want some good tech metrics to track.
We have two metrics groups here: the software delivery throughput and software delivery stability.
Throughput measures the speed of making updates of any kind to the product in production, whether those updates were planned or a response to bugs and failures. Throughout includes:
- Change lead time, or how long it takes for a new change in the product to be successfully deployed to production.
- Deployment frequency, or how often new changes are deployed to production. Notice, I say production, not Dev, QA or UAT environments.
- Failed deployment recovery time, or the time it takes to recover from a failed deployment and get things running again.
Stability measures the likelihood deployments unintentionally lead to immediate, additional work. Stability includes:
- Change fail rate, or the percentage of deployments that cause failures in production and require fixes.
- and Rework rate, or the percentage of deployments that were not planned but were performed to address a user-facing bug in the product.
With that, what DORA found is that there are four distinct clusters of teams and organizations based on their overall performance: elite, high-performers, medium, and low.
What this graph shows us is that elite performers, basically, the creme de la creme of technology, can deploy to production multiple times per day, have only about 5% failure rate and when it fails they are able to fix it within an hour, and they can generally get a new change or functionality deployed within a day.
On the low end we have teams and organizations who struggle with 40% failure rate that can take up to a month to resolve. They are able to deploy to production not more often than once a month, and it may take as long as six months.
Throughout and stability is correlated. It means that the less frequent your deployments, the higher would usually be the fail rate. This doesn’t mean that one cause the other, but that these just seem to be going downhill or uphill at the same time.
The researchers ask: Which is better, more frequent deployments or fewer failures when deploying?
“There may not be a universal answer to this question.” they answer.
But a very interesting finding, and one of the key recommendations from DORA here is that the best teams are those that achieve elite improvement, not necessarily elite performance.
Take that all the people who think retrospectives are useless. Retrospectives is basically your key factor of success longterm. Ha!
As DORA states: Improving should be more important to a team than reaching a particular performance level.
To summarize this section, DORA provides some pointers on how to use the findings from the software delivery performance metrics.
1️⃣ They recommend gathering cross-functional teams for the product you want to improve and running a value stream mapping exercise. This will allow the team to identify impediments and build an improvement plan. Here’s a useful link from DORA: https://dora.dev/guides/value-stream-management/
P.S.: I was looking around to see if there is something you can follow, like a guide for this, but I couldn’t find any consistent resources for running a value stream mapping exercise. So I decided to make my own guide, as usual. Stay tuned for when this product becomes available in my store.
2️⃣ Dedicate capacity to this improvement work! It means that you need to set enough of the teams’ time aside for just this work and prioritize it as well. It shouldn’t be an afterthought when the team has time, or something similar.
3️⃣ Build a practice of continuous improvement. Take an iterative approach to these changes.
Platform engineering
The next section I want to discuss here is platform engineering. So what is it actually?
Let me read a few key points from DORA report:
The automation in platform engineering typically includes things like setting up the environments for the new application, database,s, tests, build, and deployment.
Basically, the report is trying to see whether having a platform designed to facilitate developer work is impacting the overall organizational performance. And the short answer is ‘Yes’.
Platform engineering is putting the focus on the user and their experience, which was identified as a key factor in improving organizational performance. The user in case of a platform is the developer, so it’s an internal user, not the end-user of the application.
As the report states:
When a platform was present, developers felt more productive individually and as a team. Though, the numbers aren’t exactly impressive: 8% and 10% improvement in comparison to no platform users. The overall organizational performance only saw an increase of 6%.
And it seems that this increase in productivity came with some drawbacks: throughput and change stability decreased by 8% and 14(!)%.
One of the conclusions in DORA is that this is typical of any transformational effort and that after the initial early gains, teams are experiencing some challenges with the implementation. DORA still has a positive outlook on this and believes that in the long run, the productivity gains have more potential.
The importance of continuous improvement is seen here again as any transformational effort will go through the “j-curve”, and the productivity gains will stabilize over time.
The key finding in this section is the impact platform engineering has on developer independence.
This is very important because it removes that frustrating hand-off process that many teams have to go through to get their stuff out into production. I’ve seen this so many times, especially in bigger organizations, where teams are done with their work but have to wait for weeks to see it deployed with no ability to influence the speed since the “enabling team” is busy with other stuff. While the stakeholders are blaming the developers for being too slow. This has an extremely negative impact on the morale as well as trust.
Anyway, as the DORA report states:
I personally believe that the more important aspect of this is not just productivity, but as I mentioned just now, it’s the positive impact this will have on the team morale, motivation, and trust between the team and the stakeholders.
Let’s quickly touch on the two downsides of platform engineering: the 8% decrease in throughput and 14% decrease in change stability. DORA has a couple of hypothesis why this happens and what to do about it.
Decrease in throughput, according to DORA hypothesis, is probably due to the potential increase in the number of hand-offs between systems that give more chances for additional wait time to be introduced. Another hypothesis, is that teams may be required to exclusively use the platform, even when it’s not fit for purpose.
So…
Decrease in change stability indicates that the change failure rate and rate of rework are significantly increased when a platform is being used. In addition, this is linked to higher levels of burnout.
Ouch, that doesn’t sound good!
Though, DORA report does state the following:
Some hypothesis and possibilities as per DORA are:
- Developers have higher confidence when pushing the changes to platform since it would be easier to recover if something goes wrong. This in fact has a positive effect since it encourages experimentation and reduces the fear of failure.
- Second hypothesis, is that the platform isn’t very effective at ensuring the quality of changes. Or maybe the teams don’t use the platform’s full capabilities to test their code by prioritizing throughput instead. This can be remediated by encouraging the right behaviours of prioritizing quality first.
- Third possibility is related to the people who answered the survey in the first place. It’s probably the teams who are already struggling with instability and burnout who try to build platform to help them overcome these challenges. This is what elevates the negative impacts in the survey results. So it’s more of a correlation, than causation.
In the end, while, as DORA states:
There are a couple of actions that DORA report encourages teams to take to balance the trade-offs when embarking on a platform engineering initiative:
- Make developer independence and self-service capabilities the most important factors when deciding what platform functionality to prioritize.
- Monitor the instability of the application changes and discover the underlying reasons for decreases in stability if they occur.
Developer experience
The next section of the report is called Developer Experience, and is directly related to user-centered mindset.
In short, putting user needs first has an overwhelmingly positive impact on organizational performance and developer experience.
One of the points that I found extremely satisfying personally is this (where “they” mean “developers”):
I literally wrote “based” in my notes next to this paragraph.
I often see developers who believe that meetings and planning, and all of this work, is just not valuable and a waste of their time because they have real work to do (real work being building functionality). But it’s the collaboration with the customers that is essential to their end success and even for creating better working environment.
I can read the whole section here out loud, but that would not be very good content. Instead, I’ll just summarize.
When organizations put their focus on the user needs they build better products. We already know that. This in turn has positive effects on the team itself, their morale and motivation, and it reduces burnout. It gives the teams better sense of accomplishment and progress. This also elevates the product quality in on itws own.
But DORA also states that there is another path to success:
As shown in this graph:
However, the authors warn us that building products based on assumptions as we only focus on throughput
Other points mentioned in this section are:
- Stable priorities, that don’t shift constantly, lead to increases in productivity and decreases in employee burnout. Who would have thunk!
- Getting and incorporating user feedback provides clear sense of direction and allows for better prioritization. Hello, Sprint Reviews!
- Cross-functional collaboration is essential for building high-quality products, because even the most talented developer doesn’t build software on their own. So yes, you actually have to talk to other people. And yes, if you think you are the smartest person in the room, you should leave the room.
- Good documentation practices are essential for long-term success. BUT
I was disappointed to find a huge misunderstanding of the Agile Manifesto in the DORA report, so I have to talk about it briefly.
Here is what the report says:
🤦♀️
And while they state in the second paragraph that “comprehensive documentation” may be a phrase standing in for unhealthy practices….
it seems that they clearly misunderstood the main premise of the manifesto that states “while there is value in the items on the right, we value the items on the left more”.
Dear DORA authors. Yes, the Agile Manifesto means that comprehensive documentation, especially unhealthy practices that encourage writing documentation for bureaucratic reasons to never be used by anyone, should NOT hinder the delivery of working software.
It doesn’t mean that we should not have comprehensive documentation.
This also applies to all other Agile values.
(rant over)
Anyway, I highly recommend any leader or developer to read this specific section of the report in detail. This is basically what all Agile practitioners have been nagging you about for years!
Since you don’t believe us, read it in a report created by the more technical people, who “undertand your dev work better”.
Leading transformations
This last section covers some important points in relation to bringing all the findings to life in your team and organizations. It gives some steps to take and practices to implement.
Here’s one of the highlights:
The report also touches on transformational leadership, which I really like. It basically talk about all of those important points we usually talk about when we describe great Scrum Masters or just Agile leaders.
The report states that they saw 9% increase in employee productivity when transformational leadership is increased by 25%. Again, the numbers aren’t exactly impressive, but it’s good to see.
Other positive effects of transformation leadership include:
Transformation leadership can help enable high performance through the adoption of technical and product-management capabilities and practices. More specifically, here are some actions and practices that help:
- delegating authority and autonomy to teams.
- providing the metrics and business intelligence needed to solve problems.
- creating more focus on value delivery as opposed to feature delivery.
Plan for improvement – it’s part of your job!
chapter: To summarize…
I’m gonna start wrapping up this video here.
This of course was a lot of information. And there is even more in the report itself. If you have time, I recommend reading the report in it’s entirety.
I found some great nuggets of advice in there, though many have already been on my radar for years. But I also found some confirmation of what I believe to be true. Now backed by some data.
Maybe it can help me in encouraging teams to implement some of the practices that will lead to their success.
In the summary, DORA states:
And what are your thoughts on the report? Did you understand some parts differently? How can you use the findings to help your team?
