Category Archives: Data Science

Business Experimentation

Imagine for a moment that you want to implement a new sales initiative that you think will transform your business. The problem is, you’re not too sure if it’d work.

You decide, prudently, that maybe a pilot test would be good: let’s roll out the initiative to just a small subset of the company, the pilot group, and see how it performs.

If it performs well, great, we roll it out to the rest of the company. If it performs badly, no drama – we simply stop the initiative at the pilot stage and don’t roll it out to the rest of the company. The cost of the pilot would be negligible compared to the full implementation.

After consulting with your team, you decide that your pilot group would be based on geography. You pick a region you know well with relatively homogeneous customers, and whom are extremely receptive to your idea.

You bring your idea to your boss, who likes it and agrees to be the project sponsor. However, he tells you in no uncertain terms that in order for the initiative to go beyond a pilot, you need to show conclusively that it has a positive sales impact. You have no doubt it has, and you readily agree, “of course!”

Knowing that measurement is a little outside your area of expertise, you consult your resident data scientist on the best way to “show conclusively” the your idea works. He advises you that the best way to do that would be through doing an A/B test.

“Split the customers in your pilot group, the region you’ve picked, randomly into two,” your data scientist says. “Let one group be the ‘control’ group, on which you do nothing, and the other be the ‘test’ group, on which you roll out the initiative on. If your test group performs statistically better than the control group — I’ll advise you later on how to do that — you know you’ve got a winning initiative on your hands.”

You think about it, but have your doubts. “But,” you say, “wouldn’t that mean that I would only impact a portion of the pilot group? I can’t afford to potentially lose out on any sales – can’t I roll it out to the whole region and have some other group, outside the pilot, be the control?”

Your data scientist thinks about it for a moment, but doesn’t look convinced.

“You can, but it wouldn’t be strictly A/B testing if you were to do that. Your pilot group was based on geography. Customers in other geographies won’t have the exact characteristics as customers in your pilot geography. If they were to perform differently, it could be down to a host of other factors, like environmental differences; or cultural differences; or perhaps even sales budget differences.”

You’re caught in two minds. On the one hand, you want this to be scientific and prove beyond a doubt the efficacy of the initiative.

On the other hand, having an initiative that brings in an additional $2 million in revenue looks better than one that brings in an additional $1.5 million, due to having a control group you can’t impact.

Why would you want to lose $500,000 when you know your idea works?

What do you do?

A Culture of Experimentation

Without a culture of experimentation, it’s extremely difficult for me to recommend that you actually stick by the principles of proper experimentation and go for the rigorous A/B route. There’s a real agency problem here.

You, as the originator of the idea, have a stake in trying to make sure the idea works. Even though it’d have just been a pilot, having it fail means you’d have wasted time and resources. Your credibility might take a hit. In a way, you don’t want to rigorously test your idea if you don’t have to. You just want to show it works.

Even if it means an ineffective idea is stopped before more funds are channeled to an ultimately worthless cause, for you it really has no benefit. Good for company; bad for you.

In the end, I think it takes a very confident leader to go through with the proper A/B testing route, especially in a culture not used to proper experimentation. It’s simply not easy to walk away from potential revenue gains through holding out a control group, or scrapping a project because of poor results in the pilot phase.

But it is the leader who rigorously tests his or her ideas, who boldly assumes and cautiously validates, who will earn the respect of those around. In the long run, it is this leader who will not be busy fighting fires, attempting to save doomed-to-fail initiatives.

Without these low-value initiatives on this leader’s plate, there will be more resources that can be channeled to more promising ventures. It is this leader who will catch the Black Swans, projects with massive impacts.

I leave you with a passage from an article I really enjoyed from the Harvard Business Review called The Discipline of Business Experimentation, which is a great example of a business actually following through with scrapping an initiative after the poor results of a business experiment:

When Kohl’s was considering adding a new product category, furniture, many executives were tremendously enthusiastic, anticipating significant additional revenue. A test at 70 stores over six months, however, showed a net decrease in revenue. Products that now had less floor space (to make room for the furniture) experienced a drop in sales, and Kohl’s was actually losing customers overall. Those negative results were a huge disappointment for those who had advocated for the initiative, but the program was nevertheless scrapped. The Kohl’s example highlights the fact that experiments are often needed to perform objective assessments of initiatives backed by people with organizational clout.

Can you imagine if they decided not to do a proper test?

What if they thought, “let’s not waste time; if we don’t get on the furniture bandwagon now our competitors are going to eat us alive!” and jumped in with both feet, skipping the “testing” phase?

Or what if the person who proposed the idea felt threatened that should the initiative failed  it would make him or her look bad, and decided to cherry pick examples of stores for which it worked well? (An only too real and too frequent possibility when companies don’t conduct proper experiments.)

It would, I have little doubt, led to very poor results.

And now imagine if this happened with very single initiative the company came up with, large or small. No tests, just straight from dream to reality.

Disastrous.

But unfortunately in so many companies just the case.

How to make better decisions using Opportunity Cost

The cynic knows the price of everything and the value of nothing.
— Oscar Wilde

Opportunity cost can help you make better decisions because it helps put your decisions in context. Costs and benefits are framed in terms of what is most important to you at the time of the decision.

Every time we make a decision involving mutually exclusive alternatives, we will always be subject to this thing called “opportunity cost“.

Opportunity cost is the cost you pay for choosing one alternative over the others. But this cost isn’t “cost” in the regular sense of the word. It is the benefits of the next-best alternative that you have given up.

I hope you’re still with me here. But even if you’re not, don’t worry. I’ll give more examples below.

The concept of opportunity cost illustrated in under 60 words

You are given a choice between two pieces of fruit: an apple and an orange. You can choose only one. By choosing one, you give up the other. If you choose the apple, your opportunity cost would be the enjoyment of the orange. And if you choose the orange, your opportunity would be the enjoyment of apple.

The concept’s that simple. You give up the enjoyment of the orange when you choose the apple, so you “pay” for the apple by giving up the opportunity to enjoy the orange. So far so good? Good.

Let’s shake things up a bit.

Opportunity cost can be for indirect costs too

The scenario. Say a man, we’ll call him Man A, comes up to you and demands from you a glass of orange juice. If you don’t give in to his demands within the next five minutes, he’ll spill permanent ink all over the shirt you’re wearing, which just so happens to be your favourite. Unfortunately for you, you don’t have any orange juice or oranges on hand.

Suddenly another man, Man B, whom you had once given a banana, comes up to you and serendipitously offers to return the fruity favour. Not knowing what type of fruit you like, he offers you a choice of two fruits, an apple and an orange, from which you can take one. You grab the orange, thank him, and quickly make some orange juice for Man A, saving your favourite shirt from certain doom.

Let’s say that in normal times you would pay $1 for an apple and only $0.80 for an orange. Without Man A, the guy who threatened to spill ink on you, you’d have most definitely gone for the apple because it’d have been a better value. But because you knew of what would happen if you didn’t get the orange juice to Man A in time, you opted for the orange.

Opportunity cost is context-sensitive. You gave up the opportunity for an additional $0.20 in value (the difference between the apple and the orange) for the opportunity to save your shirt. Very smart.

Money isn’t everything: Applying opportunity costs to decisions not involving money

Thinking about the opportunity costs also helps us to think about value beyond price (as illustrated by the example above). And sometimes when price isn’t a factor at all, this can be especially important.

The scenario. Imagine facing the decision of painting a wall in your room green or blue. The paint of both colors cost the same, and both look equally good. So you consider flipping a coin and letting chance determine the colour.

But because you’ve learned about opportunity cost, you ask yourself, If I paint my wall blue, what do I give up? And if I paint my wall green, what do I give up?

After giving it a little think, you realise that by painting your wall blue, you’d probably not be able to hang your favourite poster because the colours wouldn’t match. Some of the furniture you had previously picked out would also have to be given up because they didn’t match the blue colour scheme either.

A green wall, on the other hand, would suit the poster and the furniture you picked out just fine. With your knowledge of what you had to give up if you chose the blue paint, you decide to go for the green.

Remember, if cost was the only consideration, you might not have gone for green. If colour preference was the only other consideration, you still might not have gone for green either. It was only after you considered everything in context, figuring out what had to be sacrificed (the poster and the furniture you had picked out), that you could make an informed decision.

Score one for opportunity cost.

Opportunity cost and time

Opportunity cost can also be applicable to time. If you’re stuck doing activity A, chances are you won’t be able to do activity B at the same time.

The scenario. Suppose you’ve just learned how to do your taxes. You estimate that it will take up about two hours of your time if you did it yourself.

Your cousin, who happens to love doing taxes, offers to do it for you for $50, with a free tub of ice-cream (she’s dropping by the supermarket and there’s a two-for-one special).

If you rejected your cousin’s offer, you’d save yourself $50. But it’d cost you two hours of your time doing whatever you wanted, the actual work of doing your taxes, and a tub of ice-cream.

If you took your cousin’s offer, apart from getting your taxes done for you, you’d a free tub of ice-cream. Of course, you’d have to pay her $50 — that’s the cost.

Depending on how much you valued your time, and how much you valued money, I’d say it’s a tough call. If you felt that an hour of your time was worth only a dollar (and two hours of your time being worth two dollars), it would probably make sense for you to do your taxes if you felt neutral about it (i.e. you didn’t hate doing it).

If you felt that an hour of your time was worth $50 on the other hand, letting your cousin do your taxes would probably make pretty good sense, since you’d essentially be getting back $100 worth of time for an expense of $50.

But we’re cold rational beings, and “feelings” of how much our time is worth just doesn’t cut it. So how do we find out how much our time is really worth? Again, here comes opportunity cost to the rescue.

Using opportunity cost to find the monetary value of your time

The scenario. Suppose you earn on average $25 per hour doing freelance work. Let’s say you’ve got more than enough jobs to go around, and that any free time you have could be used to your work. If you didn’t have to do your taxes, you’d be working on your freelance gigs, earning $25 per hour.

The estimated monetary value of an hour of your time would then be $25, which is the amount you’d earn if you had put that hour to work.

So, carrying on from the previous example, if you had given up your cousin’s offer you’d save yourself $50 but be giving up $50 in lost paid work (that’s $25 x 2 hours) and a tub of ice-cream. If you do the math that’s a negative return.

But if you took up your cousin’s offer, assuming you used those two hours you saved to work, you’d break-even and get a free tub of ice-cream.

Everything else being equal, there’s no reason why you shouldn’t be taking up your cousin’s offer.

Think about all your decisions using opportunity costs. And before making a decision, ask yourself these questions:

  • If I choose alternative A instead of alternative B, what am I giving up?
    • Now that I know what I’m giving up, what are the consequences of giving that up?
  • Are there any hidden benefits or costs I’m not seeing? Anything in terms of:
    • Time;
    • Energy;
    • Money; or perhaps
    • Intangibles?

With practice, thinking in terms of opportunity costs and benefits forgone or sacrificed will come naturally to you. And you’ll start looking not at the price of things, but the value of everything.