Be Careful What You Measure

As a kid, I used to read a lot of science fiction. Often these stories followed some design pattern or other, one of my favorites were stories patterned on The Monkey’s Paw. In the Monkey’s Paw, the main character in the story receives a magical object (the monkey’s paw) that enables him or her to make three wishes. The holder of the monkey’s paw makes a wish, the wish is granted, but the wish has unintended (and usually very bad) consequences. For example, in one story, the wisher asks for riches, and someone he loves dies and he’s left with the insurance money. Ultimately, the wisher regrets what he asked for.

Focusing on a single metric is almost always a Monkey’s Paw. #

A very common mistake is to pick a metric and wish for it to improve. It’s great to have a single number to rally around, but it’s almost always the exact wrong thing. Focusing on a single metric will often have serious consequences. More often than not, focusing on two, or even three, measures won’t make much of an improvement.

Monkey’s Paw: I wish for user growth. #

A great measure is how quickly you are growing your user base. This is a sure sign of success, right? I wish my users would double month over month for the foreseeable future. We’re going to focus totally on growth!

Monkey’s Paw honors your wish and [poof] you are spammer. You make something that abuses your users’ address book. You bring people on, but they leave just as fast. You see explosive growth, but no one stays around. You start building a bad reputation, and your company is dead in 60 days.

To avoid the curse of the paw, you want to also focus on engagement. You need to identify a measure that tells you how people are using your service. Yes, your user base is growing month over month, but are the users deriving any value from it? Is there any indication that they will be there three months, or two years, from now? You want both growth and engagement.

When wishing for user growth also track user average minutes on the site.

Monkey’s Paw: I wish for more orders. #

Your product is resonating with consumers and it’s starting to fly off the shelves. You wish for orders to increase tenfold over the holidays. That would make your year, right?

Monkey’s Paw strikes again: Orders increase twenty times, but the average order total goes down and returns go up. You are now processing more orders with a lower margin and you’ve got four times the returns than before.

To keep the paw at bay, you want to determine the quality of the orders. Are your promotions driving down the average order size? Perhaps you’re attracting a different customer base that’s more inclined to return product? Ensure that, as orders increase, you’re increasing the right kind of orders.

When wishing for more orders, also track order size and orders percent returned.

Monkey’s Paw: I wish my users were more engaged. #

You have a steady stream of new signups coming in the door; every month your numbers are doubling. But you aren’t seeing them stick around, and many of them never come back. You improve the user experience and create subtle incentives for them to stay. Yes, the users are engaged! Your community is thriving, and everyone is talking about what an amazing service you provide.

But wait—what about revenue? Have you just created a haven for freeloaders? Have people stopped signing up for the paid service? They’re engaged, but a much smaller percentage are converting from free to paid.

The key to avoiding the paw here lies in linking engagement to revenue. You need to measure conversion to paid and probably how long they stay a paid customer. You want to track purchases related to engagement, so you can determine how long and how engaged users are before they convert. Once you identify the point of conversion from free to paid, you’ll have better insight about how engagement can drive revenue for your company.

When wishing for more engagement also track conversion to paid.

Monkey’s Paw: I wish for more product downloads. #

Your product needs to be downloaded and evaluated before a user converts from free to paid. Over the past year, you’ve seen revenue increase on par with increased downloads. That’s the metric to drive! You increase downloads—you sell more product! Yes, I wish for more downloads!

Six months into your campaign to increase downloads, you see that revenue is no longer keeping pace with downloads. What happened? Why aren’t your downloads converting to paid?

The mighty Monkey’s Paw has struck again! You’ve increased downloads without qualifying the integrity of the download, and you’ve been gamed. With downloads, you want to track unique visits to make sure the downloads aren’t all coming from a couple of bots. You may also want to look at the referrers and user locations to identify which ones convert to paid. These additional metrics will help you gauge the quality of the downloads and prevent you from being gamed by high download numbers.

When wishing for more product downloads also track US downloads and downloads from IP addresses only seen once.

Multiple measures avoid the Monkey’s Paw. #

Focusing on a single metric will lead to certain disappointment. In setting a goal, identify elements that will ensure a well-formed result. A single measure can be gamed and is not an indicator of success without qualifying events that support the overall goal. Identify your goal and develop the story around how you envision that success. That story will reveal various metrics to track in support of your goals. And you will be a wisher with no regrets.


Now read this

Learning to Program

The Basic programming language is a really nice abstraction for a physical computer. As a language it has the notion of an instruction pointer (line numbers) and a stack pointer (using GOSUB). The entire language description fits on a... Continue →