Coder Legion - Recent posts tagged continuous-improvement

Common mistakes in DevOps metrics

Tue, 18 Nov 2025 12:41:44 +0000

Metrics are crucial to DevOps and Continuous Delivery as part of the process of continuous improvement. However, you must balance collecting and displaying data with being flooded with information. You need to decide what data to collect and what smaller set of data you pay attention to at any time.

If your car had a dashboard that displayed every metric it collects through the engine management system, there'd be no space for a windscreen.

Early cars only featured an ammeter on their dashboard, which measured the electric current between the battery and voltage regulator. This was important as it told you the charging system was working. There was no speedometer. Cars could only achieve a top speed of 35 miles per hour, and the suspension discouraged traveling at this speed. It wasn't necessary to measure velocity.

In a modern car, no dashboard space is given to an ammeter (though a battery light will illuminate if there's a problem). But, you will find a speedometer in almost every car. The current dashboard design reflects the development of cars and the broader system they operate in. Engines are more powerful, suspension systems are better, roads are generally smoother, and more cars are on the road. Attitudes towards safety have also changed.

Similarly, the metrics you collect and show will change over time as you respond to differences in your team and organization.

As you create and evolve your measurement system, there are some common mistakes you should avoid.

Ignoring the data

The first problem with metrics is that you expend great effort collecting them, but they're not always used. This happens even in organizations that regularly review the numbers.

Your data needs some process that generates insights. You can turn what you discover from the information into theories you use for experiments. The experiments should then provide new data to start the cycle again.

The only good reason to collect DevOps metrics is to learn more about your work and find ways to improve it. If you collect data just in case, the chances are it will be misused or not used at all.

Activity bias

There are 4 levels of measurement, with activity often being the earliest and easiest to collect. You can usually track activity in near real-time. The result of the activity is not typically available as an output or outcome-based metric until some later date.

Measurement level	Software example	Heating example
Activity	Lines of code	Power consumption
Outputs	Features per week	Heating element temperature
System output	Lead time	Room temperature
Outcomes	User value	People are comfortable

Activity metrics are usually a built-in feature of your existing tools, so they're already available. The problem is not all activity represents progress. Some activities might even reduce output and result in worse outcomes.

You can use activity metrics to predict future changes to outputs and outcomes. To do this, you must continually test the relationship between your activity metrics and your related output or outcome metrics.

If you only measure activity, you get a lot of motion but no progress.

Tracking too much at once

The number of metrics you collect and display can increase, often rapidly. Before long, you've filled your dashboard with charts, and you don't know what's important and what isn't.

You need to keep your tracked metrics lean, current, and relevant. When a chart is no longer helpful, you should remove it from your dashboard. You should also consider whether the metric still needs to be collected and retire it if you don't have a good reason to track it.

If you already have a dashboard, open it, and for each chart ask, "What would I do differently if this number went up or down?". Revisit this question frequently, deleting any chart where you don't have an answer.

Your metric set should focus on key long-term output and outcome metrics, with dashboards showing short-term metrics across all categories in your current improvement efforts.

Jumping into tools

Data visualization tools like Microsoft Power BI, Tableau, or Google Data Studio are among the most useful software products you'll have in your organization. Many business tools have a grid or text-based interface, but data tools come stacked with colorful animated charts.

It's easy to be distracted by creating an engaging dashboard. If you don't begin with metric design, you end up with lots of pleasing dashboards that have no impact on your daily work. You need dashboards and charting tools to help you make sense of the information, but design the metrics first.

It's better to start in low-fidelity to collect meaningful metrics. It's okay to start with a simple spreadsheet or even a whiteboard. After you work out which measurements are helpful to your team and organization, start automating the collection and creating slick displays.

If you invest too much time creating a stunning dashboard, you'll find it harder to delete charts when they're no longer required.

Standardization

Some organizations try to replicate the success of a high-performing team by making other teams follow the same process. This is rarely successful, as each team works on different problems and has different skill levels. Just as the process and practices need to be context-specific, so do the metrics.

You should address metrics as part of your continuous improvement activities. For a team with long lead times, you measure batch size, queue times, and the time work spends in each state. These metrics are unsuitable for a team whose primary problem is quality.

This requires the data, insight, theory, experiment cycle where you look at the information you have about the problem you want to solve, form a theory about what might improve the issue, then run an experiment to test your idea.

The metrics you collect also signal to the team what's important at the current time. You often see improvements simply because your metrics communicate that you care about some aspect of software delivery.

Relying on eyeballs

You should create a simple dashboard to display the metrics you're tracking for experiments. This should be displayed on an information radiator so everyone on the team can see the data.

However, if you only react to data when someone looks at a dashboard, you'll crowd the dashboard with too much information and miss key events. Long-term metrics you keep to track progress over time will seem as important as the short-term metrics you use to improve your software delivery.

The key to resolving the problem of retaining long-term metrics without cluttering your dashboard is to put in place a monitoring and alerting process for the data. Automated alerts should tell you if the metrics cross a threshold, and you could use anomaly detection to tell you if something interesting happened.

With metrics tracked automatically, you can remove them from the dashboard to free up space.

Rewarding performance

If your team is working to increase their deployment rate, it can be tempting to incentivize them with a reward if they achieve daily deployments. This approach to motivation leads to poor outcomes. A team might let other critical work slip to achieve the goal - not to cheat the system but because you made daily deployments more important than anything else.

In the landmark book Punished By Rewards, Alfie Kohn explains that attempting to manage people with incentives leads to long-term harm for your organization. Hundreds of studies have found that people do worse work when offered rewards.

Using metrics to create a competitive atmosphere, either for individual performance, comparison of different teams, or to gamify the workplace (where you introduce game elements as a form of "fun" competition), all lead to trouble.

Competition conflicts with what you really need in your organization: collaboration. If you work on the assumption that people want to do good work, you'll find you don't need to use rewards or punishments to get them to improve their performance.

Conclusion

The 5 DORA metrics and the SPACE framework provide pre-built, balanced ways to measure software delivery performance. (There used to be 4 DORA metrics, but an additional reliability metric was added.)

A good set of metrics will mix leading indicators to predict performance with lagging indicators that check the accuracy of predictions. Measurements should span activity, output, system output, and outcome categories.

We cover DevOps and Continuous Delivery metrics in detail in our white paper on measuring Continuous Delivery.

Our DevOps Insights in Octopus give you better visibility into your company's DevOps performance by surfacing insights based on 4 key DORA metrics. These metrics help you qualify the results of your DevOps performance, and gain insights into areas for improvement.

Whatever you measure, you need to constantly refine your metrics to ensure they remain useful to your team and organization. Ideally, the metrics you collect are relevant to a specific experiment you're running to test a theory.

If you use metrics well, you amplify performance and learning as you seek to become one of the elite performers in software delivery.

Improving culture with the capability culture cycle

Tue, 28 Oct 2025 10:30:43 +0000

This post is a companion to a talk I gave at the first Octopus virtual conference, SHIPPED23. I shared how DevOps capabilities help you change organizational culture by working against your team’s internal reputation. The capability culture cycle focuses on small actions under your control, creating conditions for change in the wider organization. You can watch the full talk here.

The capability culture cycle

Cultural change is a considerable challenge, especially for individual contributors who depend on influence rather than authority to bring change. That doesn’t make it hopeless. You can create an environment where change is more likely by changing how you work. Instead of directing change, as a leader might, you’ll work on the signals you send and their impact on people outside your team.

As a technical contributor on a software development team, you can make small changes that build momentum for a broader impact. The capability culture cycle uses the DORA (DevOps Research and Assessment) model as inspiration. The model contains many ideas for things you can do that directly improve your chances of moving the needle on culture. With the model as inspiration, you’ll change how you work, build a reputation, and collaborate with others in the organization to move external constraints.

The DORA capability model

When it comes to software delivery, no research effort comes close to the rigorous and long-running DORA program. Their capability model features practices linked to software delivery performance, well-being, and organizational outcomes. You can view the latest version of the DORA Core model on the DORA research site.

Culture can be intentional. If you have a strong vision, communicate it clearly, and back it up with consistent behaviors, you can build a solid long-term culture. More often, culture evolves over time and mostly by accident. It’s easy to confuse people when your rewards send a different signal to your values.

But signals aren’t just sent from leaders to staff. This point is crucial to grassroots cultural change. Many teams have constraints imposed on them because the organization lacks confidence in them. When people outside your team try to tell you how to work or need to sign off on what you’ve done, they’re telling you they don’t trust you to do the work. This often isn’t your fault; your predecessors may have done things that justify the fear.

You can transform this situation by demonstrating professionalism and ability. This requires a certain mindset combined with a set of technical capabilities. You need to show you care about the business outcomes, and you need practices that improve your software delivery performance.

Working in small steps, you can grow confidence in your ability to execute and use these gains to convince people to flex constraints. The goal is to plant the seeds of a high-trust, low-blame culture.

How things got bad

You may recognize this story. You’re working on an e-commerce platform, and most of your trading takes place in December and January. People buy things to gift to others in December and fill the gaps in their wish lists in January. For some organizations, this can represent 80% of their annual sales.

One time, a decade ago, someone deployed a new version of the platform 2 weeks into December. The software version had a problem that impacted sales. Nobody will ever forget the atmosphere of panic and dread while the developers fixed the problem. A week after the fix, management etched a new rule into stone. There will be an annual change freeze from the start of November to the end of February.

While the change freeze protects sales, it has some nasty side effects. You can’t stop development for 4 months of the year, so the developers continue to make changes. They create a large batch of changes that all go live in the first deployment after the change freeze. It’s a practical certainty there will be a big problem in this batch, and it’s equally sure it will be hard to unpick it when so many changes could have caused it.

Every March, when you deploy this large batch, all the organization’s fears are confirmed. “Thank goodness we didn’t deploy this in December,” people say as you try to track down and fix the problem. But if you deployed the software more often, there wouldn’t be a large batch. It would be lower risk, and it would be easier to fix any problems.

The idea behind the capability culture cycle is to reverse this process. You’ll use the cycle to design small actions that rebuild trust. You’ll replace the blame culture with one that learns and improves. This healthier culture has proven more effective in increasing reliability, even for safety-critical industries.

Let’s see how it works.

Stage 1: Delivery

Stage 1 of the cycle focuses on the deployment pipeline. In Continuous Delivery, a pipeline is everything that happens from the moment you commit a change, right through to the deployment of that change to Production and letting users see your work.

Everything that happens before you commit code is part of your creative design process. You think about the problem you’re solving, and you come up with a solution. Human ingenuity has a significant impact on this stage. However, after you commit the change, your brainpower doesn’t have transformational power over what happens next. After the commit, you should be looking at how to automate the whole process.

The DORA Core model describes technical practices that apply to your pipeline. Things like:

Continuous Integration
Database change management
Deployment automation
Test automation
Test data management

You’ll find further ideas by exploring the research more deeply, as the DORA Core only includes the most well-established findings. There’s a relationship between many of these technical capabilities. For example, if you find it hard to automate integration tests, better test data management will make it easier.

To apply the model, use your retrospectives or continuous improvement process to identify an area that could be better. Then, look at the research to identify potential practices that will help you improve. You don’t have to limit yourself to the research; you can experiment with your process to see what enhances it. The main benefit of the model is the things listed are statistically likely to work if you get good at them.

The capabilities in the model require practice. You can’t pick up test automation on Monday and see the positive results by Friday. You need to increase your skill level to a point where you get a benefit. Sticking with something long enough to get good at it is a crucial mindset shift.

Because it takes time to see the outcomes change, it helps to use a more responsive set of measures to see if things are getting better. This is where the DORA metrics for software delivery performance will help, as you’ll see improvements sooner. Software delivery performance is also a reasonable indicator of your team’s reputation, where it overlaps with cultural factors.

So, in stage 1, you look at ways to improve your deployment pipeline’s speed, reliability, and frequency.

Stage 2: Batch size

After one or two changes, you should see a demonstrable improvement in your software delivery performance. In response to this, you should be able to reduce your batch size. When your deployment pipeline is slow, you have little choice but to gather many changes before pushing them through a painful deployment process. Stage 1 is about making it safer and easier to deploy software so you can do it more often.

When you decrease batch size, you’ll find each deployment has fewer problems. With less concurrent change, you reduce the chances of clashes creating side-effects and it’s easier to identify the source of any problems. You’ll also be able to resolve issues faster, as the fix won’t be stuck in a large batch of work. Crucially, you’ll be able to get your work to users sooner and get feedback that influences what you work on next.

You can get feedback by talking to users and from contextual metrics. If you introduce a change to increase completion rates for an e-commerce checkout process, you can review the impact of your change to see if it works. You might transform completion rates, or it may be a minor improvement. Knowing the outcome tells you whether to keep going or bank the win and move on to the next problem. If you reduce completion rates, you can step back and return to the previous checkout process to prevent further losses.

By connecting your work to valuable outcomes, like software delivery performance and contextual metrics, you demonstrate professionalism. People outside your team will see how you’re improving your ability to execute (because you’ll shout about it). They’ll also notice you’re paying attention to business concerns. This is crucial for the next stage, as it builds trust.

Stage 3: Building trust

All of the most dreaded software delivery methods have something in common. They’re heavyweight because they’re designed for low-trust environments. Most steps are attempts to control the people doing the work because the organization doesn’t believe they can work without close supervision.

The worst part is that a heavyweight process limits a team’s ability to do good work and slows their cadence. It forces teams to work with large batches, which are more likely to go wrong when they’re deployed to Production. This results in adding more checks and balances in a self-defeating loop. The first 2 stages in the capability culture cycle are designed to stop this loop and reverse it.

As you prove you can deliver software in small batches with minimal problems, you help people outside the team grow their confidence in your ability. When the organization views the software team as professionals, they notice how well they handle a Production issue instead of claiming the problem should never have happened. The process is sometimes gradual, but the technical capabilities will increase trust.

When you see the signs that the people outside your team feel safe in your hands, you’re ready for stage 4.

Stage 4: Collaborate to remove barriers

Stage 4 is where you ask the business to move with you. You’ve demonstrated your ability to do great work, and the business can see the results. Now, you ask them to relax a constraint or remove a barrier. Get them to make your life easier so you can do even better.

Ask them to simplify the change approval process or invest in some tooling. Choose something you can take back to stage 1 of the cycle to keep building on your excellent work so far.

If you’ve tried to make changes before, you know they’re easily dismissed. This time, though, you can share your software delivery performance charts showing the improvements and data on the impact made by code changes. You’ll discuss any Production incidents and remind people how you handled them so much better than before.

Repeat the cycle

On your first time around, you might not make the biggest dent. You may improve slightly, build a little trust, and get the business to move a short distance. All progress is good. It’s crucial to keep building on this by repeating the cycle.

The changes you get in stage 4 are fuel to take into stage 1 to power the improvement engine and get another full rotation.

Culture bubbles and the values oasis

Repeating the capability culture cycle creates the right conditions for change. This may be enough in some organizations to grow into complete organizational change, or the effects may be localized. You can increase the chance of broader cultural change if the organization’s leaders support it and work actively to make it happen.

Where you don’t ignite broader changes, you may end up with a culture bubble or a values oasis. Understanding the limitations that come with the territory can help you ensure you still benefit from all your hard work.

Values oasis

Will Larson coined the term values oasis. It describes a situation where your leader uses their personal capital to create a non-conforming pocket of culture in the organization. A values oasis only works while the leader remains. If the leader moves on, the team is suddenly exposed to the true organizational culture, and, very often, they all leave shortly after.

A team’s long-term success depends entirely on the leader remaining in place, but they’re often sheltering the team from the rest of the organization and you can’t do that forever.

Culture bubble

A culture bubble is a pocket of culture in an organization that differs from the rest. It may be intentional or situational. For example, it might be a leader’s deliberate strategy or the result of office design, where one department has a separate space that encourages different behaviors.

Culture bubbles can be good or bad. The bubble is simply the result of one part of the organization having different circumstances to the majority.

Here’s an example I’ve seen repeated many times. A team that didn’t fit into headquarters may escape an unhealthy facilities team. They can design their own space, which often means it better suits how they work. The team can be a little louder than they’d be in the open-plan spaces in the main building, which makes it easier for them to collaborate.

The tools you use can make a big difference to culture. Something as simple as being able to use the walls without the fear of an all-staff memo can result in a significant cultural shift.

But the positive outcome only applies to the folks in the bubble. Back at HQ, teams work in forced silence and can’t use the walls. This is also a bubble created by gradually normalizing what an outsider would believe to be terrible working practices.

The danger with a cultural bubble is that it might burst. When a team returns to the main office, the bubble collapses, just as the change of leader can dry up the values oasis.

Positive steps forward

The limitation of a values oasis or culture bubble is its fragility. That doesn’t mean they aren’t worthwhile.

As long as they last, a group of people has improved well-being, and the organization benefits from these teams performing better. This is certainly a good enough reason to accept the fragile nature of these cultural mechanisms. It’s also possible to grow the cultural change, potentially to encapsulate the whole organization if the leaders have the crucial epiphany that connects the dots on the capability culture cycle.

Even if it’s a long shot, it has to be better than continuing to do things as they’ve always been done. The monotonous march towards irrelevance brings no more security than a culture bubble. For the individuals, what they learn in the bubble will help with their career, whereas the drudgery of how things are done around here never does.

Recapping the cycle

The capability culture cycle has 4 stages, and you should repeat them to amplify the impact.

Delivery: Improve your deployment pipeline by adopting capabilities.
Batches: Reduce the size of batches to deploy more often with less risk.
Trust: Build trust between the technical team and other areas in the organization.
Collaborate: Ask the organization to move with you by doing something to make your work easier and better.

Whether you grow a bubble, an oasis, or transform your whole organization — the capability culture cycle helps you make progress continuously.

You can watch the full video on YouTube.