Are You Building the Right Thing: The Metrics That Measure How Fast You Learn

In the last article, we made the case for the technical signals that tell you whether your teams are getting better at building software. Near the end, we admitted those signals have a limit. They tell you whether you’re building software well. They don’t tell you whether you’re building the right software.

This article is about that second question.

How do I know my teams are building the right thing?

It’s a harder question, and a more uncomfortable one. Deployment Frequency, Lead Time, Change Failure Rate, and Mean Time to Restore measure your delivery engine. A strong DORA score means that engine is fast and reliable. It doesn’t mean the engine is pointed at the right target.

That’s the trap. A fast delivery engine aimed at the wrong thing doesn’t save you. It just means you build the wrong thing faster, and at greater cost. And with AI making teams quicker still, that risk is getting bigger, not smaller.

The real question isn’t whether you can know the right thing up front. You usually can’t. The real question is whether your team can find out, fast and cheap, before you’ve over-invested in the wrong one. That question, technical metrics answer well.

The Right Thing Is Discovered

Most of the time, nobody knows the right thing up front.

A well-known Standish Group study found that 45% of features in shipped software are never used at all, and another 19% are used only rarely. Most of what teams build doesn’t earn its keep. And it isn’t because the teams were careless. It’s because the right thing is genuinely hard to know in advance.

The right thing is discovered, not specified. You build a version, put it in front of real users, watch what happens, and adjust. Then you do it again. The teams that build the right thing aren’t the ones with the best up-front plan. They’re the ones with the fastest, cheapest way to find out they were wrong.

So “are we building the right thing?” is almost the wrong question to ask your teams. They can’t answer it with certainty, and neither can you. The better question is this: how quickly would we find out if we weren’t? That question has a measurable answer.

What Uninformed Progress Looks Like

When a team has no real learning loop, progress looks great.

Teams mark features done. The roadmap moves. The status reports are green. From a leader’s seat, it looks like a team executing well.

We call this uninformed progress. The team is moving, but nobody actually knows if they’re moving toward something customers want. They’re building on an assumption made months ago, and they won’t test it until the big release. By the time the truth arrives, the wrong thing is already built.

This is especially dangerous in a rewrite or a rebuild. A rebuild feels safe because the destination seems known. You’re just rebuilding what already exists. But a rebuild is actually a rare chance to ask a question almost nobody asks: is this ten-year-old feature still needed at all? Teams that skip that question carry a decade of unexamined assumptions straight into the new system.

Uninformed progress is exactly what the next four metrics are designed to expose.

Four Metrics for How Fast You Learn

Here are four metrics we watch to tell whether a team can actually find out it’s wrong. Two measure how fast the learning loop turns. Two measure what the loop reveals.

How fast the loop turns

Batch Size

This is how much scope a team builds before a real user can react to it. A small batch is a thin slice of a feature. A large batch is optimism hiding uncertainty. The larger the batch, the bigger the assumption. Ship small, and you find out you took a wrong turn in a week. Ship big, and you find out after a quarter, or after a two-year rebuild. Batch Size and DORA’s Deployment Frequency move together: if you deploy often, you can ship small; if you deploy rarely, every release carries more risk and more assumption. The leader’s question: are we learning in small bets, or betting big on confidence?

Time to Feedback

This is how long it takes from starting a piece of work to learning something from a real user. It builds directly on DORA. Lead Time for Changes measures how long from code committed to code in production. Time to Feedback picks up where Lead Time ends and keeps the clock running until reality has a chance to respond. Teams that learn fastest aren’t necessarily smarter. They just let reality into the room sooner. The leader’s question: how long can we stay wrong before we notice?

What the loop reveals

Feature Usage

This is the share of what you’ve shipped that people actually use, measured from instrumentation in the product, not from opinion in a meeting. It’s one of the most direct measures of whether you’re building the right thing. A feature that ships and gets no traffic is often the wrong thing, made visible. Those Standish numbers stop being abstract statistics and become a list of specific features you can name. Use is proof of value, not shipping. The leader’s question: do we care enough to check whether anyone actually uses what we built?

Rework Rate

This is how much recently shipped work gets redone soon after it ships. In the last article, churn was a code-health signal. Here it’s the same measurement read through a different question. When teams rework fresh code heavily right after release, that’s rarely a coding problem. It usually means the team didn’t understand the problem before they built. Some rework means the team is listening. Endless rework means they’re guessing. The leader’s question: are we refining what we shipped, or rebuilding it?

A Real Example

A few years ago we had a team that rebuilt a mobile point-of-sale app for a payments company.

The old app was more than ten years old. It was iOS only, which forced the company’s customers to buy expensive Apple hardware when cheaper Android devices were right there. It had no automated tests and a tangled, complicated codebase. Customers were leaving for competitors.

The company’s own estimate to rebuild it was two years. Two years before a single customer would see anything new. Two years of watching customers leave. And there was strong pressure to make the rebuild a perfect copy. Many people insisted the new app had to match the old one feature for feature before anyone could use it.

We didn’t do that. Instead, we built the simplest complete point-of-sale we could think of, for the simplest possible customer: a cash-only donut shop. That first release was ready in about four months, not two years. Then a taco truck, a slightly more complex quick-serve case. Then up the ladder, one customer type at a time, toward a full sit-down restaurant.

Each step was small enough to put in front of a real business. An actual restaurant ran each new version for a week and handed back what worked, what broke, and which options they missed. That is Batch Size and Time to Feedback working together.

The old app carried more than 700 configuration options, built up over a decade. The instinct was to rebuild every one of them. But because real customers were using the new app at every stage, we could see which options actually got used. Between 200 and 300 of those 700 turned out not to matter enough to build before the final release. We didn’t argue about them in a conference room. The usage data settled it.

None of this would have worked without a fast, reliable delivery engine underneath. The team integrated their work dozens of times a day and could release on demand. The new code was simple and carried close to 5,000 automated tests that ran hundreds of times a day. So when feedback said change course, changing course was cheap. It was an adjustment, not a rewrite.

The rebuild shipped about 25% faster than the company’s own two-year estimate, and across more than 1,000 features built, real customers hit only about 20 minor defects. But the thing we think about most isn’t a number. The staged approach bought us a conversation with the client. Every release, we sat down together and asked what the next kind of restaurant actually needed. We weren’t guessing in a room. We were deciding with evidence.

Why This Matters Even More with AI

In the last article, we argued that AI amplifies the engineering system you already have.

AI is collapsing the cost of building. A team with good AI tooling can produce a working feature far faster than it could two years ago. That sounds like pure good news. It isn’t.

When building gets cheap, building the wrong thing gets cheap too. The constraint moves. It used to be that building was the hard part. Now the hard part is knowing what is worth building at all. AI makes the learning loop the whole game. If your team can generate features faster than it can find out whether anyone wants them, you don’t have a productivity win. You have a faster way to fill your product with dead weight.

A Practical Next Step

You don’t need to interrogate your teams about whether they’re building the right thing. They can’t prove it, and pressing them will only produce confident guesses.

Ask the loop questions instead. At your next review, ask how big the last few releases were, and how long it took before a real user touched them. Ask which shipped features have the usage to prove they were worth building, and which ones nobody can vouch for. Ask whether recent work is being refined or rebuilt.

If your team can’t answer those questions, that is the finding. It means the learning loop isn’t instrumented, and uninformed progress is the most likely thing happening right now.

One caution. These metrics measure whether your team can learn fast. They don’t make anyone act on what’s learned. A fast loop feeding a roadmap nobody is willing to change is just a quicker way to be ignored. The metrics are the easy part. Listening is the leadership part.

Because building the right thing was never about getting the plan right up front. It’s about how quickly you’re willing to find out you were wrong, and how cheaply you can do something about it.

This post comes from our software engineering practice, which specializes in refactoring application architecture and optimizing delivery to support modular teams, faster feedback, and continuous value delivery.