I just read an article on ZDnet on the dismal failure of ‘Big Data’ that was so bad I don’t even know where to start. The author states that economics is a “big data” profession (why–what makes a profession “big data” or not?), and then goes on to say that because big data hasn’t been able to solve all the world’s economic ills, it has failed.
Surely, a Big Data profession such as the study of economics over the past 150 plus years would by now be refined and almost scientific in its precision, especially since these days we have as much compute power as an economist might need, not to mention even more data to analyze. But it’s not even close.
This much data (enough to be called “big data”) wasn’t available until the recent past, so why drag 150 years of history into it? It isn’t like we’ve been working with big data related to economics for the past 150 years. The technologies, skills and mindsets related to handling this much data are still very young.
What is more, I don’t really get what he means by the failure of big data. “Big data”, by definition, simply means data that the data cannot be (or is difficult) to handle using legacy techniques involving just a single computer. By saying that big data has failed, it implies that it can succeed. But how? What’s the thing that differentiates the success and failure of “big data”?
If success means solving all the world’s economic problems then by golly, it has failed. But that’s like saying that if a pair of running shoes promises to help you run with less pain but doesn’t help you win the Olympics has failed. Certainly, there are limits to what big data can do, but saying that it has failed doesn’t make sense.
Possible successes — if you will call it that: Web analytics (where data is so easily grown because of the ease of data collection and the number of people and actions that can be measured) and politics (which was, actually, driven quite a bit by web analytics).
BONUS possible success: sports, specifically baseball (not big data per se, but it’s an interesting read and it’s about data, so there.
Donn, fragments 