The Evolutionary Advantage of a Resistance to Change

I was just thinking about organisational change and pondering over how our natural tendency to change is to resist it, when this thought popped up: if resistance to change is so hardwired in our brains, it must serve some purpose — but what?

One of the premises of evolutionary theory is this: if something survives (be it an organism or perhaps even an idea), its survival can be attributed to certain traits or characteristics that help it survive. These traits or characteristics are developed/evolved over time: as those that have these traits survive, and those that don’t die, more and more of the population will have these traits, eventually becoming a “norm”.

Such as a natural tendency to resist change.

At least that’s the theory.

So, let’s make a provocative statement here (inspired by Edward de Bono’s po): Po, a resistance to change isn’t necessarily a bad thing, despite all the bad press it’s getting. In fact, it may be good.

I think that before we get all gung-ho about the next big thing and bashing old ways of doing things, we should think about why things are the way they are, especially if they have been that way for a long time.

Old ways of doing things are, generally speaking, antifragile. They’ve withstood the test of time and have a decent history of working. Good ideas tend to remain good ideas, and the longer they survive the better they tend to be; bad ideas, on the other hand, are discarded as soon as they’re found out (a caveat: those that do manage to survive for long, however, tend to be the most dangerous — bad things are made worse when they’re not known to be bad. Think insidious. Think CFCs.)

So if you have an old way of doing things, it may not be the best way, but it’s likely a way that generates decent results, enough for it to have lasted as long as it has. The moment a better way of doing things is found and tested to work, the old way is discarded. But until then, the old way is the best way.

If people weren’t resistant to change, on the other hand, good ideas wouldn’t have the time to spread. We’d be flitting from one idea to the next, discarding great ones and embracing bad ones in equal measure.

So a resistance to change isn’t all that bad. It’s the way things should be. The incumbent has earned its right on the throne, and the onus should always be on the challenger to prove its worth.

So it does worry me if people rush into new ways of doing things without having redundant systems in place, just in case. I mean, let’s not be too hasty in burning bridges.

If there’s going to be a process change, have the old process remain in place until the new process is proven stable. Depending on what the process is, a few iterations (or days, or months) is most certainly needed. Give it time to prove its antifragility.

When automobiles were first introduced, horse-drawn carriages didn’t disappear overnight. Concurrency. Then obsolescence.

Resist, consider, then change. Carefully.

The Case for Modelling

Last week I wrote about the need for experimentation over models. Models are generally too abstract and far afield from reality that it’s hard to get any accurate answers.

If you want to find out if an intervention works, try it. Do experiments on it. Carry out pilot projects and see what happens.

Don’t model just model something to see what’s “likely to happen”. What’s “likely to happen” in your model is going to be highly dependent on the assumptions you make, which in turn are likely to highly dependent on your view of the world. We tend to see what we expect to see, which can make modelling a particularly self-referencing exercise if we’re not careful. Experiments help us see the world as it really is.

But that’s not the full story. Don’t discount modelling just yet.

Joshua Epstein makes a great case for modelling in what is one of the best essays on modelling I’ve read so far. He has one particularly salient point about models that I’d never thought of before: that building models is done by everyone all the time; it’s just that people don’t build explicit models all the time.

By building an explicit model, listing down assumptions and having all the numbers laid out in front of you, at least you know what’s going into your thinking. Not building explicit models, on the other hand, doesn’t mean you’re not using a model, as Epstein explains:

It is just an implicit model that you haven’t written down.

The choice, then, is not whether to build models; it’s whether to build explicit ones. In explicit models, assumptions are laid out in detail, so we can study exactly what they entail. On these assumptions, this sort of thing happens. When you alter the assumptions that is what happens. By writing explicit models, you let others replicate your results.

Overall, the points Epstein makes on modelling are practical and well thought out, and certainly worth the read. Models are, in a nutshell, great at testing assumptions and straightening out thinking before any real decision is made, even if that decision is to go ahead on a pilot project.

Some thoughts on the thinking behind building things

I’m going to write a bit about my thinking process whenever I build things, whether it’s websites, VBA applications, or financial models. I’m not too sure if this is going to be a “oh that’s so obvious why is he telling me this?” piece, but if there’s one thing I’ve learned, one person’s normal behaviour may be another person’s exceptional one.

And I know you’re exceptional. So let’s get on with it.

My thinking process whenever I’m making things people use (i.e. goes into production) goes like this:

First, I ask myself (or the project champion/sponsor/requester): is this a one-off, or will there be many iterations?

If it’s a one-off, I can do away with making things beautiful and/or flexible. For one-off builds (again, whether it’s a webpage or financial model), most things can be hardcoded (e.g. unchanging interest rates, cell references and the like) and documentation doesn’t need to be too detailed (trust me on this though: documentation should still be done. People have a habit of waiting till the day after you forget what you did to ask you how you did it.)

If there are going to be many iterations, then I ask myself, what’s likely to change?

I once built a financial model containing over 30 options from which business users could pick and choose to determine what affected the final number. The model didn’t start of with all the options straight off the bat, and was in fact supposed to be a one-off. But after hardcoding and then manually changing what users needed multiple times within the same day (even though they “were quite certain” on what they wanted early on), I realised the flexibility of options, despite the longer initial development, was worth doing. I built the options on a very granular level so users could say, “I want option #1, #5, and #29” and have the affected numbers come up immediately.

Focusing on what’s more likely to change, and ignoring those that won’t, will allow you to save plenty of time. In that same model, I didn’t bother with ensuring formulas didn’t get broken when headings changed, because I knew they weren’t going to. (But I have certainly done this before; in some applications I’ve written, the source data had headings manually typed in, and almost every other week some variation of what was basically the same thing would get put in. I used regular expressions to ensure anything that looked like a header was treated as a header.)

Who and what is this report for?

Depending on the needs of the request, you could very well end up spending hours on something that should take only minutes. I have encountered many times someone coming to me for numbers in which I thought details were needed (or would be useful). I’d spend far too long extracting and formatting really granular data, when all the person needed was a “ballpark” figure.

And when you’re sending that consultant who is requesting for your company’s sales figures the information he needs, make sure you’re not giving him more than he needs. Does he really need to see the invoice numbers? Or customer IDs? Or costs?

Designing for maintenance

A few years back while taking the train I saw an engineering student’s lectures notes on “designing for maintenance”. I’m not from an engineering background, but because I happened to be mulling about on what software to use for a website I was developing, the concept of designing for maintenance struck me like Eric Clapton’s fingers on a guitar string.

Beautiful. Smooth. And oh-so-right.

The two main pieces of software that I had been considering were pretty distinct. One had a very developer-friendly, Linux feel. The other had a very polished, user-friendly Apple feel. The former catered to my nerd needs, while the latter catered my aesthetic aspirations (an aside: the very word ³aesthetic² is surprisingly pleasing to the eye and ear).

Though at first I was kept in a 50/50 bind between the two, after getting exposed to this idea of “designing for maintenance”, I realised that I really needed to go for the more polished and user-friendly one. Though I was helping to develop the website, I wasn’t going to be running it or doing the day-to-day maintenance of the site.

I think that maintenance of software (including spreadsheets) is something that’s missed by far too many people. If you know you’re not always going to be around to maintain the software, make sure it’s easy-to-understand and that everything’s well documented (even if it’s in an e-mail). Keep in mind that it isn’t all about you, but rather about the end-user, too.

Well, that’s about what I have for now. Would love to hear your thoughts on this. If you have any.

Business vs. IT

I was reminded today in a book I’m reading on visual analytics that the purpose of any analytical project is ultimately to make better decisions. Coming from a mixed business and IT background, I have had my fair share of IT vs. Business conundrums.

With my IT hat on, I’m always thinking about efficiency, optimisation, ease-of-use, and resource management. What questions are being answered, what problems are being solved, or what the ROI (to a certain extent) isn’t always on the forefront of projects (though almost always who’s the requester is).

With my business hat on, I’m always thinking about what business questions to solve, what information needs I have, and just plain old “getting the answer”, which IT isn’t always the most willing to help retrieve. I don’t care about how long IT takes (so long as its done yesterday) and how much effort they need to put in or how optimised the process is, I just want my questions answered so I can make better decisions. Isn’t that what technology-driven analytics is supposed to do

But, coming from a mix of business and IT backgrounds, I know the problems both sides face. IT needs more emphasis on understanding business needs, while business needs to understand IT constraints.

My current role has me more as a “business” person, and I don’t know if that’s a blessing or a curse. A blessing in that I’m given a lot more face time in business and finance discussions, allowing me an almost voyeuristic view of how the business makes its money. A curse in that IT treats me as any other “business” user who’s request trigger-happy and who doesn’t understand the IT side of things (and therefore who should be ignored most of the time).

It’s tough convincing IT to talk about the things that could potentially give pretty decent ROI when they don’t trust you.

“I’m one of you,” I tell them.

“Yeah, you sure are, buddy,” they reply, smiling, handing me a ticket number.

For the lack of a plan

I spent close to five hours working on an assignment that I thought would take me an hour at the most.

It was an ad hoc project, and as often the case with many ad hoc projects you can’t really tell how long they’d take till they were done. My initial estimations were based off of casual look-throughs the source data I was working on, and a rough understanding of what the requirements were.

Expecting it to be not too complicated, I jumped into the assignment without thinking too much about it.

I’ll think about it as I go along, I thought.

And going deeper into the assignment, I quickly grasped what had to be done. A plan of attack was developed on the fly, and on I went.

Almost five hours later, on I was still going.

What went wrong?

I think the biggest mistake I made was rushing into things. I wanted to get the project out as soon as possible, and had set unrealistic internal and external completion time estimates. This led to a complacency at the start; an escalation of worry in the middle; and a near-panic toward the end.

With a little more forethought, I might have saved myself chunks of time. Had I known the multiple files I needed to process were so similar, I’d have done it in batches instead of piecemeal. Had I known the requirements more intimately, I wouldn’t have removed data I eventually needed. And had I known how unstructured the data would be, I’d have quoted an extended timeline that would have given me more breathing space.

In an attempt go faster, I skipped what might perhaps have been the most important thing: planning.

Student loans and how the deed is infinitely stronger than the word

Interesting article giving the perspectives of three people with outstanding student loans and how they’re paying it off.

I’d never been that heavily in debt and I do sometimes wonder what I’d do. Though I cannot say for sure, I do not see myself holding off the payment of loans if I could I help it. But I probably won’t need to theorise much more as in a few years time my mortgage will kick in and it’d be interesting to see if I’d practice what I preach about paying loans off as soon as I can. I cannot say for sure if that’s what I’d do.

That’s the thing about people: they may foresee themselves doing one thing, and saying they’d do it, only to do something else altogether when they really do it. They’d say I’d buy Brand X, definitely and then just as quick go on to buy Brand Y for no other reason than that because they felt like it. That’s definitely something to think about when collecting user responses in surveys and the like when major decisions are going to be made based off of it.

User requests are like hunger pangs

“So, when is the [request] going to be ready?” he asks me, the fourth person to ask in a one-week period.

This, I think to myself, is probably real hunger.

“I’m working on it,” I reply, which means I’m waiting it out to determine how important the request really is. The moment I can confidently say it’s a valid “need”, a real hunger, I move it into my high-priority queue and start work on it.

It’s not that I don’t wish to help, but system/application/report requests have a tedency to come in hugely inflated, seemingly much more important than they really are. More a reaction to an itch than a true life-saving need it’s thought to be.

I like to think of requests that come into my queue as a type of hunger. There is real hunger: the haven’t-eaten-for-days-and-starving hunger; and then there’s perceived hunger: the after-dinner craving for Pringles hunger.

When a request is of the “real” hunger variety, no matter how long you try to wait it out it’ll always be there (and the people who are requesting it won’t let you forget it’s there!)

“Perceived” hunger requests, on the other hand, tend to go away like after-dinner cravings when you give it a little time.

One problem with giving in to these “perceived” hunger requests is that, like the afore-mentioned Pringles, once you “pop you can’t stop” – these sorts of requests tend to come one after another. And it’s difficult to know when to say no because each request isn’t really that different from those that came before it.

A precedent, once set, can bind you to a cycle of petty requests (“why did her request go through and not mine?”) for the life of the project.

So my advice is: wait and see. If it’s really important you’ll be sure to know.

Which reminds me, it’s time for supper.

Choose your customers

Seth Godin recently had a great post on choosing your customers. He’s not the first to say this, and he’ll certainly no be the last, but it’s always good to hear reminders like this.

Choosing your customers first (before doing anything to get customers) isn’t intuitive. Heck, choosing your customers isn’t intuitive (who cares who buys my product as long as they buy my product, right?) but it’s definitely one thing to look out for.

When you have a project that you personally like and believe in, and one that you want to succeed, the last thing you’ll want to do is to go out to find champions for your project without thinking about who those champions are or what they stand for.

Before you know it, you’ll have people whom you thought were on your project’s side, but who actually have their own agendas. And you may end up fighting a creature you brought to life, twisted in ways you’d never imagined.