Category: Data Science

Machine Learning and the New Racism

Scary stuff, but something I think we’re already deeply mired in: Physiognomy’s New Clothes (the new racism, courtesy of machine learning).

Reminds me of the book Weapons of Math Destruction, which also highlighted many important points about the problems with “runaway” algorithms, which not only face the danger of falling into a closed feedback loop (and thus feeding their native biases), but also where the builders of the algorithm are no longer around to ensure the algorithm’s still behaving according to theory and can no longer validate its results qualitatively.

What is more, having worked on many data science projects, I know how easy it is to build models that can be tweaked to say anything I want by just tweaking a couple of parameters.

And let’s just say that models that don’t quite agree with management’s decree don’t always see the light of day.

(Link to article above “Physiognomy’s New Clothes” via the wonderful Marginal Revolution blog, which also highlighted the fact that this was “neglected” — I personally am finding myself increasingly leaning toward the AI doomsayers. The more I know, the more I worry.)

Getting Excited About Small Data

The next few quarters for analytics in my company are, from my perspective, going to be game-changing, and I’m excited to say my team’s taking the lead on it: from machine learning and advanced visualisations to new ways of thinking about data, we’re currently taking the steps to get to what I call “the next phase of analytics”. We are a small team with big dreams.

But what I often get from friends (and some colleagues) when I tell them about the things my team is doing, though, are questions on how “big data” is playing a part in it. Specifically, how it figures in our plans for the next few quarters.

When I tell them it doesn’t, they look at me as if I just said I loved eating broccoli ice-cream: perplexed; a little disgusted; and mixed with a bit of pity on the side. (If you clicked on that link or you know that song, you might have guessed I’m doing that parenting thing.)

“Big data” simply doesn’t factor in those plans (yet). We have enough small data to worry about to even think about big data. And yet, to them small data is yesterday’s news. It’s as if small data doesn’t count; as if it’s nothing to get excited about.

But it does count. And to those who haven’t yet experienced the joys of wringing all the value out of small data, it is downright exciting.

Sure, big data has the potential to change the world, and in many cases it already has. But by and large most of the value of big data still lies in its potential.

Small data, on the other hand, has long shown its ability to change the world.

I love especially this little story from the book mind+machine by Marc Vollenwider:

Using just 800 bits of HR information, an investment bank saved USD 1 million every year, generating an ROI of several thousand percent. How? Banking analysts use a lot of expensive data from databases paid through individual seat licenses. After bonus time in January, the musical chairs game starts and many analyst teams join competitor institutions, at which point the seat license should be canceled. In this case, the process step simply did not happen, as nobody thought about sending the corresponding instructions to the database companies in time. Therefore, the bank kept unnecessarily paying about USD 1 million annually. Why 800 bits? Clearly, whether someone is employed (“1”) or not (“0”) is a binary piece of information called a “bit”. With 800 analysts, the bank had 800 bits of HR information. The anlaytics rule was almost embarrassingly simple: “If no longer employed, send email to terminate the seat license.” All that needed to happen was a simple search for changes in employment status int he employment information from HR.

The amazing thing about this use case is it just required some solid thinking, linking a bit of employment information with the database licenses.

Small data can have big impact.

So yes, I am excited about small data!

And no, big data won’t be part of our coming analytics revolution. (Yet.)

What’s Sales Reporting Governance got to do with Bribery?

I lead a Sales Operations team, and one of our objectives for this year is to establish a “sales reporting governance structure”: to ensure that the right reports/tools get developed, with the right specifications, at the right time; and, perhaps most importantly, with the buy-in by the right people.

Essentially this governance structure looks at controlling the reporting life cycle (something like this report life cycle diagram) from when a report is dreamt up in the head of one of our business partners (our “internal customers”), through to when the report reaches its EOL (end-of-life) and can be stopped.

Though you may think this is somewhat dry work, let me assure you that it’s often anything but. Conversations can be excruciating quite colourful, particularly when it comes to prioritisation and negotiating timelines.

Take for example the following conversation between one of our business partners (BP) and us:

BP: “What do you mean you can only deliver it next Friday? I need it by Tuesday.”

Us: “Sure, that can be done, but we’ll need to stop work on the other three developments we’re working on for you that are due next Monday.”

BP: “No, you can’t stop work on those. I need those next Monday, and this one by next Tuesday.”

Us: “Sure, but we’ll have to exclude the new functionalities that you’d asked for.”

BP: “No, you can’t do that.”

Us:“I’m sorry but if you’re not able to budge on re-prioritising the other work, nor reducing the scope, there’s no way we can hit the timelines you’re asking for, especially when you’re asking for this so late in the game.”

BP: “I’m escalating this. You’ll hear from my manager.”

And so on.

In all fairness though, I have to say that in my experience most managers and senior colleagues (and anyone who has worked in, or closely with, IT) tend to understand that we have to satisfy ourselves with but 24 hours a day to do all we need to do.

These sort of escalations tend to end with “the manager” having a cordial chat with us and agreeing on a workable next step forward, none of which involves us engineering more time into the day.

Establishing a Governance Structure

Having a governance structure tends to minimise “unconstructive” conversations like those above, I think largely because of a mutual trust: the business partner trusts our verdict of whether something is possible or not impossible within a specific time frame,  while we trust that they have thought carefully through their requests and won’t be changing or adding to them unnecessarily.

But the problem with establishing a governance structure is that it, well, needs to be established, which can be incredibly tricky to get going. It’s almost like an negotiating a peace deal, where both sides want the conflict to stop, but are worried what might happen the moment they lay down their arms — will the other side take advantage and strike when they are at their most vulnerable?

I will be the first to admit that it takes a leap of faith going from a world of “if I don’t shout loud enough, and often enough, nothing’s going to get done”, to one where we’re all amicably setting and agreeing on priorities, and where promised delivery deadlines are actually being met.

It also doesn’t help that from a developer’s side, without the benefit of having past projects to tune one’s intuition, accurately estimating project scope or determining deadlines is going to be difficult;  often multiple iterations are necessary before this sort of “accuracy” is achieved. What this means is that early on, chances are good deadlines are going to be missed, which doesn’t help in building trust.

After a missed deadline or two, it’s all too easy to fall back into old patterns and proclaim that the process doesn’t work.

There will also be many, especially those more used to the “free-and-easy” days of yore, who will actively fight the change, citing that it creates too much red tape and jumping through hoops to get things done.

“We need to establish our reporting as soon as possible or we’ll just be flying blind — we can’t afford to go through this process!”

But the thing is, we often can’t afford not to.

When the number of development projects are small, I have to agree that the process, this “bureaucracy”, adds little value. We could simply get on a phone call, or write an e-mail, and agree among ourselves what needs to be done and when. If the requirement changes, no biggie, we simply tweak until its perfect – there’s sufficient slack in the system that will enable us to do just that.

But problems will occur when the number of projects starts to creep up, and more stakeholders are introduced.

The Need for a Tighter Process

The first problem is that due to the higher workload, the slack in the system that allowed for changes in between a development cycle will be gone. This means that changes or additions to the original requirement will likely have to be parked until development time opens up, which could be weeks down the road.

Business partners are not going to like that. “It’s a simple change for God’s sake!”

The thing is, no matter how small a change is, it’s going to be work. Somebody’s got to do it, and that means time out from other projects, which also have agreed timelines. If we focus on that change now, it risks jeopardising the timelines for every other project down the line.

If the change is important enough, then maybe we can take time out from another project and put it into executing the change. But it needs to be agreed by the team owning the other project. Which leads nicely to the second problem.

The second problem is that everyone will have their own agendas, and everyone’s pet project will be “of the highest priority”.

What happens when Team A, B, and C all have “high priority projects” that need to be done by next Monday, and development team only has the capacity to complete one or two? Without a proper process or governance structure, can we guarantee that the project of the highest priority for the business will be one that’s completed?

In the end, more time will be spent explaining to each of the stakeholders why their project was not completed; people will be upset, and the next time they’ll just be sure to shout all the louder, and all the more frequently. More time will be spent on meetings and e-mails, people “ensuring” this and that and never really ensuring anything at all. Estimated delivery dates will be given, but nobody would trust them because they know someone else coming in with an “urgent” request would likely take priority. If it’s “last in, first out”, why should I raise a request early only to be relegated down to the bottom of the delivery pile?

This just struck me as very analogous to the concept of bribery, which I was reminded of on my reading of the book Treasure Islands by Nicholas Shaxson:

Some argue that bribery is ‘efficient’ because it helps people get around bureaucratic obstacles, and get things done. Bribery is efficient in that very narrow sense. But consider whether a system plagued by bribery is efficient and the answer is the exact opposite. [Bribery undermines] the rules, systems, and institutions that promote the public good, and they undermine our faith in those rules.

Despite any short-term drawbacks, there are plenty of longer-term benefits, not least that of supporting stronger surrounding report development structures and a generally healthier culture.

Though setting up the governance structure thus far has been tough, with plenty of push-back and many of our business partners trying to circumvent the process we have established, I think it’s one of the most important things we can, and have ever attempted, to do.

Playing Baseball without a Bat – a great example of effective statistical visualisation

Came across a very interesting and persuasive video on baseball via Kottke.org today. It’s a great example of what an interesting question, effective visualisation, and some statistical knowledge can do.

The question the video seeks to answer is the following: what would happen if baseball player Barry Bonds, who happened to play one of his greatest (if not the greatest) baseball seasons ever in 2004, played without a baseball bat?

I’m not a baseball fan, and frankly quite a number of the things that were mentioned in the video were lost on me. But I’m a fan of interesting statistics and great visualisations, and this definitely had both.

And despite having a few doubts at its conclusion (the results seem too good to be true – watch to the end!), it is convincing and definitely worth a watch if you’re either into baseball or statistical visualisations.

Business Experimentation

Imagine for a moment that you want to implement a new sales initiative that you think will transform your business. The problem is, you’re not too sure if it’d work.

You decide, prudently, that maybe a pilot test would be good: let’s roll out the initiative to just a small subset of the company, the pilot group, and see how it performs.

If it performs well, great, we roll it out to the rest of the company. If it performs badly, no drama – we simply stop the initiative at the pilot stage and don’t roll it out to the rest of the company. The cost of the pilot would be negligible compared to the full implementation.

After consulting with your team, you decide that your pilot group would be based on geography. You pick a region you know well with relatively homogeneous customers, and whom are extremely receptive to your idea.

You bring your idea to your boss, who likes it and agrees to be the project sponsor. However, he tells you in no uncertain terms that in order for the initiative to go beyond a pilot, you need to show conclusively that it has a positive sales impact. You have no doubt it has, and you readily agree, “of course!”

Knowing that measurement is a little outside your area of expertise, you consult your resident data scientist on the best way to “show conclusively” the your idea works. He advises you that the best way to do that would be through doing an A/B test.

“Split the customers in your pilot group, the region you’ve picked, randomly into two,” your data scientist says. “Let one group be the ‘control’ group, on which you do nothing, and the other be the ‘test’ group, on which you roll out the initiative on. If your test group performs statistically better than the control group — I’ll advise you later on how to do that — you know you’ve got a winning initiative on your hands.”

You think about it, but have your doubts. “But,” you say, “wouldn’t that mean that I would only impact a portion of the pilot group? I can’t afford to potentially lose out on any sales – can’t I roll it out to the whole region and have some other group, outside the pilot, be the control?”

Your data scientist thinks about it for a moment, but doesn’t look convinced.

“You can, but it wouldn’t be strictly A/B testing if you were to do that. Your pilot group was based on geography. Customers in other geographies won’t have the exact characteristics as customers in your pilot geography. If they were to perform differently, it could be down to a host of other factors, like environmental differences; or cultural differences; or perhaps even sales budget differences.”

You’re caught in two minds. On the one hand, you want this to be scientific and prove beyond a doubt the efficacy of the initiative.

On the other hand, having an initiative that brings in an additional $2 million in revenue looks better than one that brings in an additional $1.5 million, due to having a control group you can’t impact.

Why would you want to lose $500,000 when you know your idea works?

What do you do?

A Culture of Experimentation

Without a culture of experimentation, it’s extremely difficult for me to recommend that you actually stick by the principles of proper experimentation and go for the rigorous A/B route. There’s a real agency problem here.

You, as the originator of the idea, have a stake in trying to make sure the idea works. Even though it’d have just been a pilot, having it fail means you’d have wasted time and resources. Your credibility might take a hit. In a way, you don’t want to rigorously test your idea if you don’t have to. You just want to show it works.

Even if it means an ineffective idea is stopped before more funds are channeled to an ultimately worthless cause, for you it really has no benefit. Good for company; bad for you.

In the end, I think it takes a very confident leader to go through with the proper A/B testing route, especially in a culture not used to proper experimentation. It’s simply not easy to walk away from potential revenue gains through holding out a control group, or scrapping a project because of poor results in the pilot phase.

But it is the leader who rigorously tests his or her ideas, who boldly assumes and cautiously validates, who will earn the respect of those around. In the long run, it is this leader who will not be busy fighting fires, attempting to save doomed-to-fail initiatives.

Without these low-value initiatives on this leader’s plate, there will be more resources that can be channeled to more promising ventures. It is this leader who will catch the Black Swans, projects with massive impacts.

I leave you with a passage from an article I really enjoyed from the Harvard Business Review called The Discipline of Business Experimentation, which is a great example of a business actually following through with scrapping an initiative after the poor results of a business experiment:

When Kohl’s was considering adding a new product category, furniture, many executives were tremendously enthusiastic, anticipating significant additional revenue. A test at 70 stores over six months, however, showed a net decrease in revenue. Products that now had less floor space (to make room for the furniture) experienced a drop in sales, and Kohl’s was actually losing customers overall. Those negative results were a huge disappointment for those who had advocated for the initiative, but the program was nevertheless scrapped. The Kohl’s example highlights the fact that experiments are often needed to perform objective assessments of initiatives backed by people with organizational clout.

Can you imagine if they decided not to do a proper test?

What if they thought, “let’s not waste time; if we don’t get on the furniture bandwagon now our competitors are going to eat us alive!” and jumped in with both feet, skipping the “testing” phase?

Or what if the person who proposed the idea felt threatened that should the initiative failed  it would make him or her look bad, and decided to cherry pick examples of stores for which it worked well? (An only too real and too frequent possibility when companies don’t conduct proper experiments.)

It would, I have little doubt, led to very poor results.

And now imagine if this happened with very single initiative the company came up with, large or small. No tests, just straight from dream to reality.

Disastrous.

But unfortunately in so many companies just the case.

How to make better decisions using Opportunity Cost

The cynic knows the price of everything and the value of nothing.
— Oscar Wilde

Opportunity cost can help you make better decisions because it helps put your decisions in context. Costs and benefits are framed in terms of what is most important to you at the time of the decision.

Every time we make a decision involving mutually exclusive alternatives, we will always be subject to this thing called “opportunity cost“.

Opportunity cost is the cost you pay for choosing one alternative over the others. But this cost isn’t “cost” in the regular sense of the word. It is the benefits of the next-best alternative that you have given up.

I hope you’re still with me here. But even if you’re not, don’t worry. I’ll give more examples below.

The concept of opportunity cost illustrated in under 60 words

You are given a choice between two pieces of fruit: an apple and an orange. You can choose only one. By choosing one, you give up the other. If you choose the apple, your opportunity cost would be the enjoyment of the orange. And if you choose the orange, your opportunity would be the enjoyment of apple.

The concept’s that simple. You give up the enjoyment of the orange when you choose the apple, so you “pay” for the apple by giving up the opportunity to enjoy the orange. So far so good? Good.

Let’s shake things up a bit.

Opportunity cost can be for indirect costs too

The scenario. Say a man, we’ll call him Man A, comes up to you and demands from you a glass of orange juice. If you don’t give in to his demands within the next five minutes, he’ll spill permanent ink all over the shirt you’re wearing, which just so happens to be your favourite. Unfortunately for you, you don’t have any orange juice or oranges on hand.

Suddenly another man, Man B, whom you had once given a banana, comes up to you and serendipitously offers to return the fruity favour. Not knowing what type of fruit you like, he offers you a choice of two fruits, an apple and an orange, from which you can take one. You grab the orange, thank him, and quickly make some orange juice for Man A, saving your favourite shirt from certain doom.

Let’s say that in normal times you would pay $1 for an apple and only $0.80 for an orange. Without Man A, the guy who threatened to spill ink on you, you’d have most definitely gone for the apple because it’d have been a better value. But because you knew of what would happen if you didn’t get the orange juice to Man A in time, you opted for the orange.

Opportunity cost is context-sensitive. You gave up the opportunity for an additional $0.20 in value (the difference between the apple and the orange) for the opportunity to save your shirt. Very smart.

Money isn’t everything: Applying opportunity costs to decisions not involving money

Thinking about the opportunity costs also helps us to think about value beyond price (as illustrated by the example above). And sometimes when price isn’t a factor at all, this can be especially important.

The scenario. Imagine facing the decision of painting a wall in your room green or blue. The paint of both colors cost the same, and both look equally good. So you consider flipping a coin and letting chance determine the colour.

But because you’ve learned about opportunity cost, you ask yourself, If I paint my wall blue, what do I give up? And if I paint my wall green, what do I give up?

After giving it a little think, you realise that by painting your wall blue, you’d probably not be able to hang your favourite poster because the colours wouldn’t match. Some of the furniture you had previously picked out would also have to be given up because they didn’t match the blue colour scheme either.

A green wall, on the other hand, would suit the poster and the furniture you picked out just fine. With your knowledge of what you had to give up if you chose the blue paint, you decide to go for the green.

Remember, if cost was the only consideration, you might not have gone for green. If colour preference was the only other consideration, you still might not have gone for green either. It was only after you considered everything in context, figuring out what had to be sacrificed (the poster and the furniture you had picked out), that you could make an informed decision.

Score one for opportunity cost.

Opportunity cost and time

Opportunity cost can also be applicable to time. If you’re stuck doing activity A, chances are you won’t be able to do activity B at the same time.

The scenario. Suppose you’ve just learned how to do your taxes. You estimate that it will take up about two hours of your time if you did it yourself.

Your cousin, who happens to love doing taxes, offers to do it for you for $50, with a free tub of ice-cream (she’s dropping by the supermarket and there’s a two-for-one special).

If you rejected your cousin’s offer, you’d save yourself $50. But it’d cost you two hours of your time doing whatever you wanted, the actual work of doing your taxes, and a tub of ice-cream.

If you took your cousin’s offer, apart from getting your taxes done for you, you’d a free tub of ice-cream. Of course, you’d have to pay her $50 — that’s the cost.

Depending on how much you valued your time, and how much you valued money, I’d say it’s a tough call. If you felt that an hour of your time was worth only a dollar (and two hours of your time being worth two dollars), it would probably make sense for you to do your taxes if you felt neutral about it (i.e. you didn’t hate doing it).

If you felt that an hour of your time was worth $50 on the other hand, letting your cousin do your taxes would probably make pretty good sense, since you’d essentially be getting back $100 worth of time for an expense of $50.

But we’re cold rational beings, and “feelings” of how much our time is worth just doesn’t cut it. So how do we find out how much our time is really worth? Again, here comes opportunity cost to the rescue.

Using opportunity cost to find the monetary value of your time

The scenario. Suppose you earn on average $25 per hour doing freelance work. Let’s say you’ve got more than enough jobs to go around, and that any free time you have could be used to your work. If you didn’t have to do your taxes, you’d be working on your freelance gigs, earning $25 per hour.

The estimated monetary value of an hour of your time would then be $25, which is the amount you’d earn if you had put that hour to work.

So, carrying on from the previous example, if you had given up your cousin’s offer you’d save yourself $50 but be giving up $50 in lost paid work (that’s $25 x 2 hours) and a tub of ice-cream. If you do the math that’s a negative return.

But if you took up your cousin’s offer, assuming you used those two hours you saved to work, you’d break-even and get a free tub of ice-cream.

Everything else being equal, there’s no reason why you shouldn’t be taking up your cousin’s offer.

Think about all your decisions using opportunity costs. And before making a decision, ask yourself these questions:

  • If I choose alternative A instead of alternative B, what am I giving up?
    • Now that I know what I’m giving up, what are the consequences of giving that up?
  • Are there any hidden benefits or costs I’m not seeing? Anything in terms of:
    • Time;
    • Energy;
    • Money; or perhaps
    • Intangibles?

With practice, thinking in terms of opportunity costs and benefits forgone or sacrificed will come naturally to you. And you’ll start looking not at the price of things, but the value of everything.