Getting Excited About Small Data

The next few quarters for analytics in my company are, from my perspective, going to be game-changing, and I’m excited to say my team’s taking the lead on it: from machine learning and advanced visualisations to new ways of thinking about data, we’re currently taking the steps to get to what I call “the next phase of analytics”. We are a small team with big dreams.

But what I often get from friends (and some colleagues) when I tell them about the things my team is doing, though, are questions on how “big data” is playing a part in it. Specifically, how it figures in our plans for the next few quarters.

When I tell them it doesn’t, they look at me as if I just said I loved eating broccoli ice-cream: perplexed; a little disgusted; and mixed with a bit of pity on the side. (If you clicked on that link or you know that song, you might have guessed I’m doing that parenting thing.)

“Big data” simply doesn’t factor in those plans (yet). We have enough small data to worry about to even think about big data. And yet, to them small data is yesterday’s news. It’s as if small data doesn’t count; as if it’s nothing to get excited about.

But it does count. And to those who haven’t yet experienced the joys of wringing all the value out of small data, it is downright exciting.

Sure, big data has the potential to change the world, and in many cases it already has. But by and large most of the value of big data still lies in its potential.

Small data, on the other hand, has long shown its ability to change the world.

I love especially this little story from the book mind+machine by Marc Vollenwider:

Using just 800 bits of HR information, an investment bank saved USD 1 million every year, generating an ROI of several thousand percent. How? Banking analysts use a lot of expensive data from databases paid through individual seat licenses. After bonus time in January, the musical chairs game starts and many analyst teams join competitor institutions, at which point the seat license should be canceled. In this case, the process step simply did not happen, as nobody thought about sending the corresponding instructions to the database companies in time. Therefore, the bank kept unnecessarily paying about USD 1 million annually. Why 800 bits? Clearly, whether someone is employed (“1”) or not (“0”) is a binary piece of information called a “bit”. With 800 analysts, the bank had 800 bits of HR information. The anlaytics rule was almost embarrassingly simple: “If no longer employed, send email to terminate the seat license.” All that needed to happen was a simple search for changes in employment status int he employment information from HR.

The amazing thing about this use case is it just required some solid thinking, linking a bit of employment information with the database licenses.

Small data can have big impact.

So yes, I am excited about small data!

And no, big data won’t be part of our coming analytics revolution. (Yet.)

The problem with running a team at full capacity

I shared this earlier on LinkedIn, but thought that it was worth sharing it here too as a reminder to myself: Six Myths of Product Development

I came across the article above while researching why a team that traditionally does great work may sometimes stumble (yes, mine). The past few weeks had been a whirlwind of activity, with team output close to or at an all time high. We were publishing and developing things left and right, and everyone was running close to capacity. It was great.

Then came an e-mail that questioned the quality of the output. Then another. Much of the great work threatened to come undone, but thankfully most made it through unscathed. We were still, generally, in a good place. But this was a wake up call. Something needed to be done.

After I explained to her my conundrum, my knowledgeable friend, Google, suggested an article from the Harvard Business Review website called “Six Myths of Product Development.”

It was a most excellent suggestion.

The article highlighted six myths or fallacies:

  1. High utilization of resources will improve performance.
  2. Processing work in large batches improves the economics of the development process.
  3. Our development plan is great; we just need to stick to it.
  4. The sooner the project is started, the sooner it will be finished.
  5. The more features we put into a product, the more customers will like it.
  6. We will be more successful if we get it right the first time.

It didn’t take long for me to realise that our problem was very likely linked to #1: I’d neglected slack.

You see, I normally tend guard slack time jealously as I know time-pressures are often a big cause of low quality output. But given the myriad of “urgent” business needs had allowed myself and the team to run too close to full capacity.

We have seen that projects’ speed, efficiency, and output quality inevitably decrease when managers completely fill the plates of their product-development employees—no matter how skilled those managers may be. High utilization has serious negative side effects… Add 5% more work, and completing it may take 100% longer. But few people understand this effect.

It’s funny how bringing down the amount of expected output may actually increase it.

(As an aside, I love point #6 – I’m a big fan of “fail fast, fail often” as I believe strongly in “the wisdom of crowds”, where we can aggregate feedback and iterate quickly, especially for early development. But it’s not always easy to get business buy-in, especially when all they see in “fail fast, fail often” is “fail”!)

 

The need for theory in prediction models

I’d like to share this wonderful quip by philosopher Robert Long, that was quoted in the (also insightful and actually pretty good) book A Richer Life by Philip Roscoe:

Let’s say that in early 2001 I formulate a theory to the effect that there is a Constant Tolkienian Force in the Universe that produces a Tolkien film every year. When Austrians complain that my theory ignores the fact that films are products of human action and not of constant impersonal forces, I reply: ‘Oh, I know that. My theory isn’t supposed to be realistic. The question is whether it’s a good predictor.’ So I test it in 2001, 2002, and 2003. Lo and behold, my theory works each year! … But unless I pay some attention to the true explanation of this sequence of film releases, I’ll be caught by surprise when the regularity fails for 2004.

The above quote relates to the “blind” prediction models that we build (maybe statistical, maybe machine learning), where “accurate prediction” is so often the aim of the game.

The problem with just going for accurate predictions, without thinking about the underlying causes or theory, is that it’s difficult to tell whether or not the predictions are flukes, or if they contain true insight that may not be so obvious to anybody or anything else but the algorithm.

Though the quote above is rather tongue-in-cheek (the “Tolkienian Force” model is really built on just three data points, way too few for statistical significance) the moral of the need for a good theory wasn’t lost on me: theory is the human ying to the technological yang. They balance and support each other, creating something much more powerful and persuasive than either would alone. When one is amiss, the other somehow doesn’t feel right either.

Another (More Selfish) Benefit of Theory

They’re actually really useful when persuading the business on accepting the results of a prediction model that they otherwise know nothing about. Without a theory, on what basis should the business believe the model? On faith?

That’s a tough sell.

And if you were the one who built the model, and they “trusted” the model because they trust you, what happens if the model fails?

Well, you fail with the model.

With a convincing theory, the model stands on its own merits. They may “trust” the model a bit more because they trust you, but if the model fails, it’s the theory that fails with it, not you.

What you do determines what you see

Author’s note: This post was originally titled “Déformation Professionnelle”, but I had trouble understanding it myself and have renamed it for easier future reference!

This post in three words: Profession -> Perception -> Truth

The following text is taken from the excellent book The Art of Thinking Clearly, by Rolf Dobelli.

A man takes out a loan, starts a company, and goes bankrupt shortly afterward. He falls into a depression and commits suicide.

What do you make of the story?

As a business analyst, you want to understand why the business idea did not work: was he a bad leader? Was the strategy wrong, the market too small or the competition too large?

As a marketer, you imagine the campaigns were poorly organised, or that he failed to reach his target audience… As a banker, you believe an error took place in the loan department.

As socialist, you blame the failure of capitalism.

As a religious conservative, you see in this a punishment from God.

As a psychiatrist, you recognise low serotonin levels.

Which is the “correct” viewpoint?

The above is also what is known as Déformation Professionnelle (what a term!) — a tendency to look at things from the point of view of one’s own profession rather than from a broader perspective.

I’m only too wary of falling into this trap, which is especially easy for me to do because my expertise lies in data and its derivatives and the scientific method , things I hold dear and believe are as close you can get to a panacea for all the world’s ills.

Which is why I often preface the ideas I share with, “if I put on my analytics hat…”, because I know not everybody will share the same view. And I respect that.

Business Experimentation

Imagine for a moment that you want to implement a new sales initiative that you think will transform your business. The problem is, you’re not too sure if it’d work.

You decide, prudently, that maybe a pilot test would be good: let’s roll out the initiative to just a small subset of the company, the pilot group, and see how it performs.

If it performs well, great, we roll it out to the rest of the company. If it performs badly, no drama – we simply stop the initiative at the pilot stage and don’t roll it out to the rest of the company. The cost of the pilot would be negligible compared to the full implementation.

After consulting with your team, you decide that your pilot group would be based on geography. You pick a region you know well with relatively homogeneous customers, and whom are extremely receptive to your idea.

You bring your idea to your boss, who likes it and agrees to be the project sponsor. However, he tells you in no uncertain terms that in order for the initiative to go beyond a pilot, you need to show conclusively that it has a positive sales impact. You have no doubt it has, and you readily agree, “of course!”

Knowing that measurement is a little outside your area of expertise, you consult your resident data scientist on the best way to “show conclusively” the your idea works. He advises you that the best way to do that would be through doing an A/B test.

“Split the customers in your pilot group, the region you’ve picked, randomly into two,” your data scientist says. “Let one group be the ‘control’ group, on which you do nothing, and the other be the ‘test’ group, on which you roll out the initiative on. If your test group performs statistically better than the control group — I’ll advise you later on how to do that — you know you’ve got a winning initiative on your hands.”

You think about it, but have your doubts. “But,” you say, “wouldn’t that mean that I would only impact a portion of the pilot group? I can’t afford to potentially lose out on any sales – can’t I roll it out to the whole region and have some other group, outside the pilot, be the control?”

Your data scientist thinks about it for a moment, but doesn’t look convinced.

“You can, but it wouldn’t be strictly A/B testing if you were to do that. Your pilot group was based on geography. Customers in other geographies won’t have the exact characteristics as customers in your pilot geography. If they were to perform differently, it could be down to a host of other factors, like environmental differences; or cultural differences; or perhaps even sales budget differences.”

You’re caught in two minds. On the one hand, you want this to be scientific and prove beyond a doubt the efficacy of the initiative.

On the other hand, having an initiative that brings in an additional $2 million in revenue looks better than one that brings in an additional $1.5 million, due to having a control group you can’t impact.

Why would you want to lose $500,000 when you know your idea works?

What do you do?

A Culture of Experimentation

Without a culture of experimentation, it’s extremely difficult for me to recommend that you actually stick by the principles of proper experimentation and go for the rigorous A/B route. There’s a real agency problem here.

You, as the originator of the idea, have a stake in trying to make sure the idea works. Even though it’d have just been a pilot, having it fail means you’d have wasted time and resources. Your credibility might take a hit. In a way, you don’t want to rigorously test your idea if you don’t have to. You just want to show it works.

Even if it means an ineffective idea is stopped before more funds are channeled to an ultimately worthless cause, for you it really has no benefit. Good for company; bad for you.

In the end, I think it takes a very confident leader to go through with the proper A/B testing route, especially in a culture not used to proper experimentation. It’s simply not easy to walk away from potential revenue gains through holding out a control group, or scrapping a project because of poor results in the pilot phase.

But it is the leader who rigorously tests his or her ideas, who boldly assumes and cautiously validates, who will earn the respect of those around. In the long run, it is this leader who will not be busy fighting fires, attempting to save doomed-to-fail initiatives.

Without these low-value initiatives on this leader’s plate, there will be more resources that can be channeled to more promising ventures. It is this leader who will catch the Black Swans, projects with massive impacts.

I leave you with a passage from an article I really enjoyed from the Harvard Business Review called The Discipline of Business Experimentation, which is a great example of a business actually following through with scrapping an initiative after the poor results of a business experiment:

When Kohl’s was considering adding a new product category, furniture, many executives were tremendously enthusiastic, anticipating significant additional revenue. A test at 70 stores over six months, however, showed a net decrease in revenue. Products that now had less floor space (to make room for the furniture) experienced a drop in sales, and Kohl’s was actually losing customers overall. Those negative results were a huge disappointment for those who had advocated for the initiative, but the program was nevertheless scrapped. The Kohl’s example highlights the fact that experiments are often needed to perform objective assessments of initiatives backed by people with organizational clout.

Can you imagine if they decided not to do a proper test?

What if they thought, “let’s not waste time; if we don’t get on the furniture bandwagon now our competitors are going to eat us alive!” and jumped in with both feet, skipping the “testing” phase?

Or what if the person who proposed the idea felt threatened that should the initiative failed  it would make him or her look bad, and decided to cherry pick examples of stores for which it worked well? (An only too real and too frequent possibility when companies don’t conduct proper experiments.)

It would, I have little doubt, led to very poor results.

And now imagine if this happened with very single initiative the company came up with, large or small. No tests, just straight from dream to reality.

Disastrous.

But unfortunately in so many companies just the case.

Forecasts are not predictions

If you have a prediction and it turns out to be wrong, then that’s bad.

But if you have a forecast and it turns out to be wrong, that’s not necessarily bad, and may in fact be good.

Let’s say that you’re the captain of a ship and you see an iceberg one mile out. Based on your direction and speed, you forecast that within a couple of minutes you’ll hit the iceberg. So you slow down and change course, averting impact.

The forecast, in this sense, was wrong, but served the very purpose it was meant to serve: to aid in decision making. Now, imagine if you were rewarded on getting your forecasts right. Changing course after revealing your forecast of imminent impact would have been a bad move.

To satisfy your objective of forecast accuracy while still not sinking your ship, you could also decide to stay on course, but slow down enough to ensure impact was still made but with minimal damage. Doing this, you would have met both the “avoid sinking” objective and your “forecasting” objective.

But that doesn’t really make sense, does it? And yet, isn’t that what we often do in sales forecasting?

 

Improving Forecasting Through Ensembles

There’s this wonderful article I want to share on building prediction models using ensembles. “Ensembles” in this case simply means the combination of two or more prediction models.

I’d personally had great success bringing several (relatively) poorly performing models together into one ensemble model, with prediction accuracy far greater than any of the models individually.

Definitely something to check out if you’re into this sort of thing.