Getting Excited About Small Data

The next few quarters for analytics in my company are, from my perspective, going to be game-changing, and I’m excited to say my team’s taking the lead on it: from machine learning and advanced visualisations to new ways of thinking about data, we’re currently taking the steps to get to what I call “the next phase of analytics”. We are a small team with big dreams.

But what I often get from friends (and some colleagues) when I tell them about the things my team is doing, though, are questions on how “big data” is playing a part in it. Specifically, how it figures in our plans for the next few quarters.

When I tell them it doesn’t, they look at me as if I just said I loved eating broccoli ice-cream: perplexed; a little disgusted; and mixed with a bit of pity on the side. (If you clicked on that link or you know that song, you might have guessed I’m doing that parenting thing.)

“Big data” simply doesn’t factor in those plans (yet). We have enough small data to worry about to even think about big data. And yet, to them small data is yesterday’s news. It’s as if small data doesn’t count; as if it’s nothing to get excited about.

But it does count. And to those who haven’t yet experienced the joys of wringing all the value out of small data, it is downright exciting.

Sure, big data has the potential to change the world, and in many cases it already has. But by and large most of the value of big data still lies in its potential.

Small data, on the other hand, has long shown its ability to change the world.

I love especially this little story from the book mind+machine by Marc Vollenwider:

Using just 800 bits of HR information, an investment bank saved USD 1 million every year, generating an ROI of several thousand percent. How? Banking analysts use a lot of expensive data from databases paid through individual seat licenses. After bonus time in January, the musical chairs game starts and many analyst teams join competitor institutions, at which point the seat license should be canceled. In this case, the process step simply did not happen, as nobody thought about sending the corresponding instructions to the database companies in time. Therefore, the bank kept unnecessarily paying about USD 1 million annually. Why 800 bits? Clearly, whether someone is employed (“1”) or not (“0”) is a binary piece of information called a “bit”. With 800 analysts, the bank had 800 bits of HR information. The anlaytics rule was almost embarrassingly simple: “If no longer employed, send email to terminate the seat license.” All that needed to happen was a simple search for changes in employment status int he employment information from HR.

The amazing thing about this use case is it just required some solid thinking, linking a bit of employment information with the database licenses.

Small data can have big impact.

So yes, I am excited about small data!

And no, big data won’t be part of our coming analytics revolution. (Yet.)