Statistics do not always tell the whole story

I’m not sure if you’ve read or heard about the recent unfavourable review of the Tesla Model S by New York Times reporter John Broder, but if you haven’t, you should.

Not because of the review itself, which was newsworthy in possibly putting a large dent in the credibility of the Model S as an “everyday car”, but because of the very interesting back-and-forth between Broder and Elon Musk (Tesla Motor’s
CEO), and the use of statistics/numbers to prove a point. And how those numbers fail to show the whole picture.

After the review of Broder’s was published, Musk provided a scathing rebuttal on his blog, going so far as to say that Broder “worked very hard to force our car to stop running”. Filled with logs and statistics he put forth a convincing argument. Broder then responded in his own blog, explaining just as convincingly how some of the facts Musk brought up missed some context.

Two things in particular stood out for me.

First, as an analyst, was how statistics and “facts” were used by Musk to refute Broder’s claims.

I am–was–a strong believer in the saying numbers don’t lie. And when Musk dug up the car’s logs, posting evidence that Broder deliberately set out to jeopardise the car’s performance in his review, I couldn’t help thinking there was no way out for Broder.

For example, Musk had remarked on this blog that Broder “drove in circles for over half a mile in a tiny, 100-space parking lot. When the Model S valiantly refused to die, he eventually plugged it in.” Damning evidence if there was one.

I thought that Musk made plenty of good points, but I also couldn’t help thinking that there was a possibility of the statistics not telling the whole story. Musk was, after all, selecting the logs/numbers he thought would back up his claim the best, and several commenters had brought up the point that he neglected mentioning anything about the battery’s dramatic power-loss when parked overnight.

Second, as a son of parents not particularly keen to experiment when trying out technology they’re not confident of, what Broder had written in his response to Musk’s rebuttal made perfect sense. His actions, though deemed by some commenters as “stupid” (read what he did in the next paragraph in parenthesis) were probably what any person in the context of unfamiliarity might do: rely on the experts (in this case the Tesla representative on the phone with him).

(Broder set out for his destination 61 miles away “even though the car’s range estimator read 32 miles – because, again, I was told that moderate-speed driving would ‘restore’ the battery power lost overnight”.)

On the “driving in circles” comment, Broder made himself quite clear on this point: “I drove around the Milford service plaza in the dark looking for the Supercharger, which is not prominently marked. I was not trying to drain the battery. (It was already on reserve power.) As soon as I found the Supercharger, I plugged the car in.” This claim was backed up by a number of commenters who themselves owned electric cars, saying that it wasn’t uncommon to circle around looking for chargers.

When statistics doesn’t tell the whole story

This reminded me of how businesses sometimes use statistics to measure employee performance, and how it might sometimes fail. Don’t get me wrong. I believe in that practice: statistics helps add an objective viewpoint to an otherwise very subjective activity. The problem is when statistics is taken at face value, with the context behind the numbers ignored, even when it’d have changed the story completely.

When we see employees “circling a parking lot” seemingly looking to jeopardise the company’s performance (e.g. not hitting budget or unproductive), we miss the bigger picture: that the superchargers aren’t prominent enough. So, employees are unproductive. Is it really their fault or is the support lacking, or is it something else?

If these “circling” employees are fired because of their poor performance, employees hired to take over these “poor performers” who come in are still going to be “circling” the parking lot because the root cause wasn’t found. And that’s not an entirely smart thing to do.