Tell me what you want to see

Caught this magnificent optical illusion on kottke.org today. I’d say that is definitely  this is worth a minute or two of your time.

Was in my “data” frame of mind when I watched this, and couldn’t help thinking that this is exactly how data works: control the content, control the angle (i.e. perception), and you can make a square block look like a cylinder.

Expensive Software and Consultants

They took our data, ran it through their software, and they got the answers that eluded us for so long.

I was told they were a big consulting company, which meant they probably had great, restrictively expensive software that could do the job. That’s why.

But I don’t buy that argument.

Great software needn’t be expensive.

I’ve lived and breathed great open-source, free technologies growing up. Linux; Apache; PHP; MySQL; WordPress; Python; R.

Are any of these free technologies inferior to their paid counterparts? In development (including data science) work, I don’t think so.

So why were they “successful”? Why could they come up with an answer we couldn’t?

My guess: they were a consulting company with less vested interest.

They came up with an answer. But would it have been better than the one we would have come up with if we were in their shoes? I don’t know.

As a consultant I’d have been much more liberal with my analyses. No matter how badly I mess up, the worst that would happen would be that my company would lose a contract. And chances are good I could push the blame to the data that was provided, or having been provided the wrong context, or information that was withheld.

When you’re part of the company, you have far more vested interest. Not just in your job, but your network, both social and professional. Consequences extend far beyond they would if you were an external consultant working on “just another project”. I’d be far more meticulous ensuring everything was covered and analyses properly done.

 

Business Implications of Analysis

“And,” she said, “we found that the more rooms a hotel has, the higher the positive rating.”

I was at NUS (National University of Singapore) in my Master’s class — listening to my peers present their analysis on the relationship between hotel class (e.g. budget, mid-scale and luxury) and the ratings of several key attributes (e.g. location, value, service) based on online reviews.

By now, having been through ten presentations on the same topic in the last couple of hours, it was clear that there was a link between hotel class and attribute ratings: higher class hotels tended to get better reviews.

But something was missing in most of these presentations (mine included, unfortunately): there wasn’t a business problem to be solved. It was simply analysis for analysis’ sake. Through it all I couldn’t help but think, “so what?”

So what if I knew that a budget customer’s standard of “service quality” was different from that of the patron of a luxury class hotel? So what if I knew that economy-class hotels didn’t differ from mid-scale hotels but differed with upper-scale hotels? So what if I knew that hotels with more rooms tended to have more positive reviews?

(And on this last point, it was a rather common “finding”: it was found that hotels with more rooms tended to have higher ratings, and presented as if if you wanted higher ratings, you might want to build hotels with more rooms; the problem of course is that larger hotels with more rooms tend to be of the higher-end variety; budget and independent hotels tended to have fewer rooms. Would the business implication then be that even budget hotels with more rooms will improve their ratings? Probably not.)

In the end the 15 presentations or so that we went through just felt like a whole lot of fluff. Sure he analytical conclusions were technically correct; statistically  sound. But so what?

It reminded me that you can be great at analysis, but without an understanding of the business, without a mindset of constantly questioning “so what does this mean — what are the implications on the business?”, all your analytical prowess would be for naught.

You make it look so easy

I’ll start with a quote I read today from the book Getting Ahead (Garfinkle, 2011) about a problem faced by people good at their craft. It made me smile because I this was the first time I’d seen it brought up anywhere and which I thought was one of those things I thought you just sucked up and lived with:

Former local San Francisco TV host Ross McGowan was negotiating a contract with his boss. He was surprised when his boss made a fairly low offer, especially considering how high his programs’ ratings were. McGowan asked why the offer was so low and his boss said, “You make it look so easy.”

Not to brag, but I think I do lots of great work (and so do many people I know), but oftentimes I make it look too easy, even when it’s not.

If you work with me, you’ll see the output of my design, programming, and execution. You see the 20 minutes that they can see but miss the 600,000 that has gone on behind the scenes preparing for just this very moment (and moments like these).

You don’t see the hours of PowerPoint deck preparation and storyline rehearsals I do for each and every presentation.

You don’t see the countless trips to the library I make getting books to hone my craft.

You don’t see the endless hours of coding I do just practicing, like differentiating the nuances of a while loop from a for loop so I can use it in my next project.

You don’t see the articles I read on metrics on sales team remuneration design so that I’m aware of potential flaws in the company’s compensation schemes and can proactively work around or advise on these when the time comes.

Easy? If giving up many aspects of life that you feel for and wish you had more time for is easy, then well, yes.

My thoughts on (sales) forecasting and predictive models

I need to have a data-dump on the sales forecasting process and forecasts.

On optimistic and pessimistic forecasting:

  • When forecasts are (consistently) too low: well-known issue that even has a name: sandbagging. You forecast lower to temper expectations. When you do get better results than the forecast you look like a hero.
  • When forecasts are (consistently) too high: quick research on Google shows that this is almost as prevalent as sandbagging. It seems salespeople are by nature over-optimistic about their chances of closing deals. My question though: if you consistently fail to deliver on the high-expectations doesn’t this dent your confidence? I’m not a salesperson, but if I was one I’d probably be a sandbagger (note: this actually reminds me of IT teams, where sandbagging is so prevalent  because of the high variability of project outcomes).
  • If the above is true, that we consistently under- and over-estimate our abilities to deliver, would a range of forecasts be a better bet? But I don’t hear sales leaders saying “don’t give me a number, give me a range.”
  • Would a range solve the sandbagging and over-optimism problem? In a way, it might, since it forces an alternative view that would be hidden should only a single number held sway.
  • Sandbaggers would be forced to say, “well yes, if EVERYTHING went to plan we might get a 20% increase in sales this month,” while the over-optimistic’ers would be forced to say, “fine, you are right that there is quite a bit of risk. A 10% drop wouldn’t be impossible.”
  • The problem with a range is that it is, well, a range. Oftentimes  a single number is preferred, especially if it’s to be communicated to other parties. It’s easier to tell a story with a single number, and its precision (note that I did not say accuracy) is seductively convincing.
  • One way around this would be to explicitly ask for a range, but at the same time ask also for the “highest probability” or “expected” value. This forces thinking about the best and worst case scenarios while giving you the benefit of a single number. And if you were tracking these forecasts, you might actually find that you can systematically take the optimistic forecasts of known sandbaggers and pessimistic forecasts of known over-optimistic’ers.

On the granularity of forecasting

  • When forecasting, the more granular the forecast the more noise you’ll find. I find it easiest to think about this in terms of coin flips.
  • A fair coin gives a 50/50 chance of being heads or tails.
  • If I flipped a coin, there would be a 50/50 chance of it being heads or tails, but I couldn’t tell you with any certainty if the next flip was going to be heads or tails.
  • However, if you flipped a coin a thousand times, I could tell you with certainty that the number of heads would be close to 50%, which is the nature of fair coin.
  • But let’s say I flipped a coin ten times. Could I tell you with certainty that the number of heads would be close to 50%? Well, no.
  • With just 10 flips (or “trials”, in statistical parlance), the probability of getting 5 heads is actually only 24.60%, which means that you have a 75.40% chance of getting something other than 5 heads/tails.
  • As we increase the number of trials, the probability of heads gets ever increasingly closer to 50%. Every additional trial reduces the variability, and you get closer and closer to what is the “nature of the coin”.
  • In sales forecasting there are occasionally times that you are asked to forecast for very specific things, so specific in fact that you might only have 10 historical data points from which to extrapolate. But with just 10 trials, what’s the chance that those 10 would fit the “nature of the thing being predicted”?
  • From Arthur Conan Doyle’s Sherlock Holmes: “while the individual man is an insoluble puzzle, in the aggregate he becomes a mathematical certainty. You can, for example, never foretell what any one man will do, but you can say with precision what an average number will be up to. Individuals vary, but percentages remain constant.
  • One way around this is to aggregate upwards. You can, for example, ask yourself “what category does this thing I’m trying to predict fall into?” and lump this with those other similar things in the same category.
  • Say you have 10 related products that have sold about 10 units each, similar to each other though not identical. Though you could attempt to predict them individually, the small sample sizes per product would give you so much variance your prediction would likely be not much better than chance. It would be better to group these products together into a single category and perform predictions on this larger category.
  • Variations/predictive noise at the individual product level cancel each other out, giving you a cleaner picture.
  • Though looking at the individual products is a precise exercise, it doesn’t add to predictive accuracy.

On Building Great Predictive Models

  • The greatest amount of time spent on developing good predictive models is often in data preparation.
  • Give me perfect data, and I could build you a great predictive model in a day.
  • A predictive model that is 80% accurate may not be “better” than a model that is 70% accurate. It all depends on the context (if this was in the business domain, we’d say it depends on the business question).
  • Let’s say I build a model that is so complex it’s impossible for others but the most technical minds to understand, or which uses a “black box algorithm” (i.e. you let the computer do its predictive thing, but you have no hope of understanding what it did, e.g. a neural network). It predicts correctly 8 out of 10 times (or 80%).
  • Concurrently, I also build a model using a simple linear regression method, which is algorithmically transparent – you know exactly what it does and how it does it, and it’s easily explainable to most laypersons. It performs a little worse than the more complex model, giving me the correct answer 7 out of 10 times (or 70%).
  • Is giving up control and understanding worth that additional 10% accuracy? Maybe, but in a business context (as opposed to a hackathon) chances are good that after the 7th time you spend an hour explaining why the model does what it does, you’ll probably want to opt for the more easily understandable model at the expense of a little accuracy.
  • Business understanding is an important aspects of model building.

Overfitting a model and “perfect” models

  • Finally, I want to talk about overfitting models. Have you heard about overfitting?
  • When we build predictive models, we build them based on past data. In machine learning we call this data “training data”, i.e. data we use to “train” our model.
  • Overfitting happens when we “train” our model so well on training data that it becomes so specific to the data used to train it that it cannot be expanded to predict new data.
  • I find it akin to learning a new language. Sometimes you get so fixated on the grammar and syntax and structure you miss the woods for the trees, that though your speech may be grammatically correct it could be awkward or unnatural (e.g. overly formal, which is often the case if we learn to speak like we write).
  • When somebody speaks to you in a language you’re just picking up using conversational language, you try to process it using your highly formalised syntaxes and grammars and realise that though you know all the words individually, when strung together they make as much sense as investment-linked insurance plans.
  • Overfitting often happens when we try to predict at increasingly granular levels, where the  amount of data becomes too thin
  • In the end the model becomes VERY good at predicting data very close to what was used to build the model, but absolutely DISMAL at predicting any other data that deviates even slightly from that.
  • If tests show you’ve got a model performing at too-good-to-be-true levels, it probably is. Overfitted models perform very well in test environments, but very badly in production.
  • Sometimes when a model performs “badly” in a test environment, ask yourself: (1) is it performing better than chance? (2) is it performing better than the alternatives?
  • If your answer to both (1) and (2) is yes, that “bad” model is a “good” one, and should be used until a better one comes along.
  • Unless, of course, that the resources it takes to carry out predictions, in terms of monetary cost, time, or both, are higher than the benefits it brings. Sometimes a simple model with above-average performance that can be run in a minute can be far more valuable than one with superb predictive performance but which has a turnaround time longer than the time in which decisions are made.
  • I know of some people who look at predictive model and dismiss them simply because they aren’t perfect; or worse, because they’re too simple — as if being “simple” was bad.
  • But models have value as long as they perform better than the alternatives. If they’re simple, quick to run, and require no additional resources to build or maintain, all the better.

So many ideas – have to expand on some of these one of these days.

How I became an analyst

I just approved a comment on one of my earlier posts, a post about my possible foray into sales. A post that, as I re-read it, brought back plenty of memories. A post that reminded me how my career as it stands now, that of data science and analytics, is quite different from what I had once thought I would be doing.

When I’d written that post in January 2009, I’d just graduated from the University of Western Australia and was actively looking for a job. But what job I was looking for I didn’t have much of a clue. Most of my peers hadn’t studied what I’d studied (a combination of business and information technology), or had the interests I had.

The Career as Journey not Destination

Back then I wished someone told me how jobs and careers worked; I wish I someone told me that a job or career is more journey than destination, and that not everybody knows outright what they want to do for the rest of their lives. And sometimes, careers are, as in my case, down to plenty of randomness.

So as these memories flooded back, I thought, why don’t I write something about how I found my current calling? Maybe it might help someone just starting out as well, someone as confused as I was. So here it is.

My Initial Career Intentions

Based on what I’d studied and what I was interested in, I had the following careers in mind:

  • Entrepreneurship (to be my own boss was something I’d always dreamed about, but doesn’t almost everybody);
  • Web-design (I’ve had a website since since 1997 — you might, but probably don’t, remember Geocities and Homestead, two free web-hosting platforms I used in the early days);
  • Programming (in 2002 I wrote a content management software in PHP that was essentially a clone of one of the leading CMSes at that time called MoveableType, and used it to power edonn.com);
  • Copywriting (I used to write, still do, a lot, and loved writing to influence (and still do);
  • Business analysis (whatever that was… the only reason I’d had it down was that in my university’s course material for one of my majors, information systems, it was listed as a probable career, and I’d scored a perfect GPA for the major, something I only found out in hindsight as I prepped my résumé);
  • Education (teaching or similar — I wanted to “change lives”); and
  • Financial planning (I’ll talk more on this later).

These were some of the options I had bouncing around in my head. And whatever I decided to do, my initial thought was that it should be related to non-profit or charitable work if possible. If I was going to be spending the most part of my life doing something (i.e. a job) it might as well be something “worthwhile”.

Limited Options

I would find out soon enough that though I might have tons of ideas of what I wanted to do, those tons would be whittled down quickly to what jobs were available. Being in the middle of an awful recession — remember Lehman Brothers and the sub-prime crisis? — many of the  jobs I thought I had a chance in were simply unavailable.

Sales(y) jobs, though, were still in abundance. These were jobs were largely commission-based, and no- to low-risk propositions for the employers: if you don’t sell, you don’t get paid. And I suppose because of the abundance of these jobs and my not quite knowing what jobs to look for, I decided to try my luck interviewing for them, finding out what they were about and convincing myself that I could do them (the post above was part of the “convincing myself” part, but to a large extent I stand by what I wrote — though I’m not “selling” a product to external customers, I do plenty of selling in terms of ideas , analyses and other “data products” to internal customers).

One sales(y) career that I had seriously considered was that of financial planning. I had taken (and enjoyed) financial planning at University, and I loved reading personal finance blogs and books. Helping others with their finances was also something I felt I could do, especially if I was doing something I knew would help them.

I interviewed with an independent financial planning firm (one I trusted and felt really comfortable with) and was conditionally accepted. I was told to go back when I had the necessary certifications, which I diligently went out to get (which set me back about $600).

But all this while I still wasn’t sure if that was really what I wanted to do. Did I really get a degree for this? A polytechnic diploma would have been sufficient, and having a degree didn’t really help. So I decided to let fate decide: if by April 2009 I hadn’t been offered a non-financial planning, non-salesy job, I would take up the position at the financial planning firm mentioned above.

I continued scouring job postings, looking out for the less sales(y) jobs, trying my luck by sending out résumés for which I felt unqualified but had great interest in. I also targeted jobs more relevant to my degree, which I would never had gotten with just a polytechnic diploma.

Then one day a job ad came along that made my small eyes open a little wider. One that made me think: I could do this; I want to do this. It was for a Business Analyst position, a position I had, until then, not really thought much about, but which had a beautiful mix of business, social science and information technology.

The technical skills that they were looking for seemed a little more than I could offer (specifically VBA programming skills), but everything else I had down to a tee. I sent in my résumé and hoped for the best. A few days later a call came, and I was told that they wanted to interview me (hooray!) That first interview went decently, and I was told to return for a technical skills test, which I passed.

And then I was told that there was a final interview. By this time I was feeling a little job-search weary, and I told myself that this was going to be the last interview I’m going for before I said yes to the financial planning firm.

And as fate would have it, the final interview went well, and I was told a couple of days later that I got the job.

On 1st April 2009 I started my career as an analyst at Future Electronics.

(Yes, April fool’s. “My job’s a joke; my career’s a joke” became a standard joke for me while I was at Future).

And that’s how I became an analyst (and not (yet) an entrepreneur; web-designer; copywriter; financial planner; or an educator. Though on this last one, I still do plenty of educating in my analytics role!)

Some asides:

  • In case you’re wondering what an “analyst” does, read my post “What do you do? I’m an analyst.
  • I’m no longer at Future, and currently work at Study Group (a higher education pathway provider), where I’ve taken a similar but more strategic analytics role – still not quite non-profit or charity, but I’m getting there).
  • As an analyst, especially in smaller companies or teams, or where analytics is more immature, I find that you pretty much define your own job scope. Sure, the job you’re hired for needs to be done, but there’s often so much more you can do with data and systems with lots of business impact, that if you’re gung-ho enough you can start side projects that can quickly become integral to how the business runs things.

The Teacher Being Surpassed by the Student

I was in class on Saturday thinking about how nice it would be if an article of mine on edonn.com was used as a piece of “teaching material” – i.e. quoted in class, or perhaps in the lecture notes.

It was then that a recollection of “some saying” hit me, something about the aim of a teacher is  to have students surpass him or her in skills. Well, a Google search has just confirmed I was imagining such a saying (apparently teachers don’t really care about students surpassing them in skill.)

But I did come across a Chinese saying that goes like this:  青出于蓝而胜于蓝 qīngchūyúlánérshèngyúlán, which literally translates to “blue comes from the indigo plant but is bluer (vivid) than the plant itself.”

Two things I learned: that the Chinese have some very nice and poetic sayings (well, I’ve actually known this a long time); and that I want to be that teacher who seeks to have students whose skills surpass mine.

I’ve always dreamed of having a team of data workers (data-analysts, engineers, scientists) working under me technically more skillful than me, even when hired they may not have been so. To build a Manchester United of data science within the company (my dear employers, are you reading this? Excited?! I am.)

As an aside: For even more dramatic effect, make that the Manchester United of the 1950-1960s under Sir Matt Busby. The one whose team was just about wiped out in a plane crash in 1958 on their way to the finals of the European Cup, only to return ten years later to win it. I love that story.

Making Measurement Count

There’s a saying I’ve heard many times that goes something like this: what gets measured gets done.

And though I completely agree with that saying, I think it misses a crucial point: that before measuring anything, we have to make sure that what’s getting done is what you want to get done.

After the army started testing for push-ups in their standard physical fitness tests, I started training for it. After they stopped testing for pull-ups, I stopped training for it.

I now do 50 push-ups without too much difficulty. I used to struggle at 30.

I now do 8 pull-ups with difficulty. I used to be able to do 12 with both hands tied behind my back.

Both were intended to act as proxies to my fitness. So, can someone tell me: am I less fit or more?

To answer that question, I propose we ask two further questions, the answers of which will then determine the answer:

    • What’s the definition of fitness?
    • Which exercise, pulls-ups or push-ups, is more closely correlated to that definition?

It would be great if fitness could be measured directly, but t can’t. So we use proxies or indirect measures such as pull-ups, push-ups, or running times to help approximate it. Sometimes these proxies make sense, but sometimes (many times) they don’t (e.g. how high or far you jump is often used as a proxy for fitness, but it’s actually probably more technique than anything else).

Proxy measures are used all the time in business as well: the performance of a business is often measured via financial proxies (revenue, costs, profit) and these often make sense – when there’s no profit for too long, companies go bankrupt and fold.

But many companies don’t stop there (nor should they). Things like staff turnover, lead times, number of customers and more are also used as proxies as well to determine if a business is “doing well”.

I do worry sometimes though, that in our quest to measure everything in order to understand where we’re at and seek “improvement” in some process or activity, we forget to ask what our true end-goal is (the formal definitions of what we’re trying to achieve), and whether or not the measurements we have in place are truly correlated to that goal.

What does it really mean for a business (or business unit or employee) to be “doing well” (in the context of this organisation)? And are the measures we have in place to measure if the above are “doing well” really correlated with what we have define as “doing well”?

Because if we don’t, we could well be working hard at improving the proxies without improving anything of real value.

KFC and the Representative Survey

I had KFC (Kentucky Fried Chicken) for breakfast yesterday. Chicken rice porridge and a “breakfast” wrap (that oddly enough didn’t seem to contain any chicken).

It was decent, and I liked it.

So when I was quite excited when I saw that the receipt had a link to an online customer satisfaction survey, for which I would get a free piece of chicken if I completed it. It was a pretty good deal, I thought.

But I couldn’t help but wonder about how useful it was to KFC.

Surely survey responses would be largely over-represented by people who like their food (and service, to a certain degree)? If I hated their food, and/or hated their service, and swore never to go back there again, what good would offering me a free piece of chicken do for me?

These are the people whom you probably most want to hear from, and yet have absolutely no incentive to complete such a survey (and in most likelihood, being normal people like us, they’d vote with their dollars and just not patronise the store again, instead of submitting feedback).

It would, in short, be far from a representative survey.

I just hope that those who are interpreting and on the receiving end of said-interpretation understand the limitations of just such a survey, and discount the very likely amplified, far-too-positive results.

And if the results are lukewarm instead of three-Michelin -stars-worthy? Then oh dear.

On Planning and Project Management

“So how long,” he asks, “do you think you’ll take to complete the project?”

“Two weeks,” I say.

Three weeks later, I’m still two weeks away from completion. What happened?

It’s not like this hasn’t happened before. It’s happened many times before.

Rolf Dobelli says it beautifully in his excellent book The Art of Thinking Clearly:

[W]hy are we not natural-born planners? The first reason: wishful thinking. We want to be successful and achieve everything we take on. Second, we focus too much on the project and overlook outside influences. Unexpected events too often scupper our plans.

The second point is particularly pertinent in my line of work. I have a pretty decent idea of how long a project will take, if all goes well and if I could dedicate 100% of my time to the project.

What I tend to forget is that life often gets in the way. There are plenty of interruptions. Other “important”, “even more urgent” projects pop-up; ad hoc requests that are quick to resolve but that just as quickly add up to significant amounts of time; regular routine work that’s not quite factored in, because they are normally quickly executed, but every once in a while require a long, dragged-out bout of firefighting.

Dobelli has a solution for better planning. He suggests

[shifting] your focus from internal things, such as your own project, to external factors, like similar projects. Look at the base rate and consult the past. If other ventures of the same type lasted three years and devoured $5 million, they will probably apply to your project, too — no matter how carefully you plan.

Which is a great case for keeping a log of project and their time to completion (do NOT trust undocumented recollections of project lengths; like childbirth, our brains tend to underestimate the amount of time and pain we go through).

Also, one more thing I would is that you resist the temptation to think that “this time is different”, and that no matter how confident you are, this time is probably not.

Trust the data.