The Metrics That Make Engineering Organizations Worse

A metric is a question you ask so often that you automate it. Choose the wrong question and the organization optimizes for the wrong answer: count lines of code and you get more code, count tickets closed and you get smaller tickets. Measuring engineering is not wrong. It just has to measure outcomes, flow and health, at team level, as signals to investigate rather than targets to hit.

Why It Matters

Metrics shape behavior more strongly than any memo, because people respond rationally to what they are judged on. A single badly chosen number can make a team:

• ship less useful work, while the dashboard shows improvement
• cut corners on quality
• stop helping colleagues, because helping doesn’t count
• hide problems instead of raising them

Good metrics do the opposite: they make problems visible early and help teams argue for the investments that would fix them.

Goodhart’s Law

“When a measure becomes a target, it ceases to be a good measure.” (Marilyn Strathern’s phrasing of the economist Charles Goodhart’s observation.)

What it means: a number is useful because it moves together with something you care about, such as lines of code with work done or tickets closed with progress. Once people are rewarded for the number itself, they find cheaper ways to move it that don’t involve the thing you care about, and the link between the two breaks. The number keeps improving; the reality behind it doesn’t. In engineering it plays out the same way every time:

Engineering versions are usually smaller but follow the same steps: pull requests split to hit a count, tests that raise coverage without checking anything, incidents downgraded so “time to restore” looks good.

Metrics That Backfire

None of these is useless as information: coverage shows untested modules, velocity helps planning. They become harmful when they turn into targets or comparisons between people.

What to Measure Instead

Measure at three levels, and read them together:

Outcomes

Start from what the work was for. Shipping is not the outcome; what changed for users, or for the teams you serve, is. Every kind of team has outcomes of its own:

Flow: The DORA Metrics

DORA (DevOps Research and Assessment, now part of Google Cloud) has surveyed tens of thousands of professionals since 2014. Its central finding: a small set of delivery metrics predicts how well the organization does, and speed and stability are not a trade-off. The best teams are good at both.

DORA’s Five Delivery Metrics

Why they beat output counts:

• They measure the team’s process, not one person. Writing code, reviews, tests, the build pipeline and releases all affect them, so nobody can push the numbers up alone.
• They keep each other honest. If you ship more often but break more things, the second column shows it straight away. Cheating on one number usually makes another one worse.
• They point to what to fix. Slow lead time usually means slow reviews, slow builds or changes that are too big. A high fail rate usually means weak tests or risky releases.
• They’re easy to start with. Your CI/CD tool already records lead time and how often you deploy. Compare with your own team’s past, not with other companies.

Health: The SPACE Framework

A team can keep great delivery numbers for a quarter by working weekends, then lose its best engineers. The SPACE framework (GitHub, Microsoft Research and the University of Victoria, 2021) says productivity has several dimensions and no single metric captures it:

The SPACE Framework

• Use at least three dimensions, so a gain in one (activity) can’t hide a loss in another (satisfaction).
• Include perception. How developers rate their tools and workload is real data, and often the earliest warning.
• Never use activity alone. It is the easiest to collect and the easiest to misread.
• Measure at the right level. Some metrics make sense for a team or a system and mislead for an individual.

A practical set for one team: change lead time, change fail rate, a quarterly developer experience survey and review turnaround time. Four numbers, three dimensions, none about one person.

The same researchers later added the DevEx framework, which narrows the focus to three things developers feel every day:

• Feedback loops: how fast builds, tests and reviews respond.
• Cognitive load: how much you must keep in your head to make a change, from unclear ownership to undocumented systems.
• Flow state: how often you can work without interruption.

Slow builds, flaky tests and unclear ownership show up here long before they show up in delivery numbers.

A Worked Example: DORA and SPACE Together

An illustration, not a real team’s data. A checkout team of six engineers feels slow, and leadership asks for proof. Instead of counting pull requests, the team picks four numbers across three SPACE dimensions and reads them for one quarter:

• Nothing here ranks a person. Every number describes how work moves through the team.
• The numbers checked each other. Faster lead time didn’t come from skipping review, because the fail rate fell at the same time.
• The survey found the cause first. Developers named slow builds before the delivery numbers showed why.

Using Metrics Without Breaking Them

• Teams, not individuals. Show the numbers to the team first, and say up front they won’t be used in performance reviews. Then keep that promise.
• Trends, not targets. Ask “why did lead time double this quarter?”, not “why is it above 2 days?”.
• Several, in tension. Pair speed with quality and health (see the table below).
• Context before comparison. A payments team and a marketing-site team should not share a leaderboard.
• Ask the team. When a number moves, the people closest to the work usually know why.
• Retire dead metrics. A dashboard nobody acts on is just a cost.

Going Deeper: Reading the Numbers Well

Look at Distributions, Not Averages

Lead time is skewed: most changes are quick, a few take weeks. Report the median (the typical change) and the 85th percentile (the slow ones), and read the slow tail by hand. That is where the process problems are.

Break Lead Time Into Its Parts

“Lead time went up” is a starting point, not a diagnosis. Split it into stages from tool timestamps (first commit, PR opened, first review, merged, deployed) to see where changes wait:

Keep Metrics in Tension

Test Yourself: Scenarios

Think through your answer, then compare.

시나리오 1

The Leaderboard Request

A director asks you to build a dashboard ranking engineers by pull requests merged per week.

정답 보기정답 숨기기

Push back, and offer an alternative.

• Why not: it rewards splitting work and penalizes reviewers, mentors, incident responders and whoever takes the hard problems. The best engineers often merge the fewest PRs.
• Ask what decision they want to make. “Are we delivering faster?” → team-level lead time, deploy frequency and change fail rate, next to outcomes.
• If it is about one person, that is a conversation for their manager, with context, not a number.

시나리오 2

Faster, but Breaking More

After moving to continuous deployment, deployment frequency doubled. Change fail rate also doubled, from 5% to 10%.

정답 보기정답 숨기기

• Do the math: twice the deploys at twice the fail rate is four times as many failed deploys.
• Check impact: if failures are small and rolled back in minutes, the trade may be fine; if customers feel them, it isn’t.
• Find the cause: missing tests, risky database changes, no feature flags.
• Fix forward: better automated checks and progressive delivery (canaries, flags), not deploying less.

시나리오 3

Three-Week Lead Time

Your team’s median lead time is three weeks, but engineers say a typical change takes two days of work.

정답 보기정답 숨기기

The rest is waiting.

• Split lead time into stages: time to first review, review rounds, CI, release window, QA.
• Fix the biggest wait first. Usual suspects: large PRs, a weekly release train, manual QA queues, slow or flaky CI.
• Typical fixes: a review response agreement, smaller changes, automated deploys.
• Show it worked with the median and 85th percentile over time.

시나리오 4

“Our Builds Are Slow”

A developer survey ranks slow builds as the top problem. How do you decide whether to invest, and show it worked?

정답 보기정답 숨기기

• Price it: build minutes × builds per day × engineers = hours waiting per week, plus context switches.
• Compare with the cost of the fix: caching, split test suites, faster runners.
• Set a target such as “median CI time under 10 minutes” and ship.
• Report both sides: the system metric (CI duration) and the perception metric (the next survey). Acting on survey feedback also makes the next survey more honest.

Key Takeaways

Measurement also decides how you handle debt and AI: see technical debt is not the enemy for pricing debt with these signals, and AI makes writing code cheaper for why output metrics are about to mislead even more.

Sources

• Andy Hertzfeld, Folklore.org: −2000 lines of code (February 1982)
• Consumer Financial Protection Bureau: Wells Fargo Bank, 2016 enforcement action
• Wikipedia: Goodhart’s law, including Marilyn Strathern’s 1997 phrasing and the original 1975 paper
• DORA: DORA’s software delivery performance metrics (the five current metrics)
• Nicole Forsgren, Margaret-Anne Storey and others: The SPACE of developer productivity (ACM Queue, 2021)
• Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler: DevEx: What actually drives productivity (ACM Queue, 2023)
• Martin Fowler: Cannot measure productivity