The problem with running a team at full capacity

I shared this earlier on LinkedIn, but thought that it was worth sharing it here too as a reminder to myself: Six Myths of Product Development

I came across the article above while researching why a team that traditionally does great work may sometimes stumble (yes, mine). The past few weeks had been a whirlwind of activity, with team output close to or at an all time high. We were publishing and developing things left and right, and everyone was running close to capacity. It was great.

Then came an e-mail that questioned the quality of the output. Then another. Much of the great work threatened to come undone, but thankfully most made it through unscathed. We were still, generally, in a good place. But this was a wake up call. Something needed to be done.

After I explained to her my conundrum, my knowledgeable friend, Google, suggested an article from the Harvard Business Review website called “Six Myths of Product Development.”

It was a most excellent suggestion.

The article highlighted six myths or fallacies:

  1. High utilization of resources will improve performance.
  2. Processing work in large batches improves the economics of the development process.
  3. Our development plan is great; we just need to stick to it.
  4. The sooner the project is started, the sooner it will be finished.
  5. The more features we put into a product, the more customers will like it.
  6. We will be more successful if we get it right the first time.

It didn’t take long for me to realise that our problem was very likely linked to #1: I’d neglected slack.

You see, I normally tend guard slack time jealously as I know time-pressures are often a big cause of low quality output. But given the myriad of “urgent” business needs had allowed myself and the team to run too close to full capacity.

We have seen that projects’ speed, efficiency, and output quality inevitably decrease when managers completely fill the plates of their product-development employees—no matter how skilled those managers may be. High utilization has serious negative side effects… Add 5% more work, and completing it may take 100% longer. But few people understand this effect.

It’s funny how bringing down the amount of expected output may actually increase it.

(As an aside, I love point #6 – I’m a big fan of “fail fast, fail often” as I believe strongly in “the wisdom of crowds”, where we can aggregate feedback and iterate quickly, especially for early development. But it’s not always easy to get business buy-in, especially when all they see in “fail fast, fail often” is “fail”!)

 

What’s Sales Reporting Governance got to do with Bribery?

I lead a Sales Operations team, and one of our objectives for this year is to establish a “sales reporting governance structure”: to ensure that the right reports/tools get developed, with the right specifications, at the right time; and, perhaps most importantly, with the buy-in by the right people.

Essentially this governance structure looks at controlling the reporting life cycle (something like this report life cycle diagram) from when a report is dreamt up in the head of one of our business partners (our “internal customers”), through to when the report reaches its EOL (end-of-life) and can be stopped.

Though you may think this is somewhat dry work, let me assure you that it’s often anything but. Conversations can be excruciating quite colourful, particularly when it comes to prioritisation and negotiating timelines.

Take for example the following conversation between one of our business partners (BP) and us:

BP: “What do you mean you can only deliver it next Friday? I need it by Tuesday.”

Us: “Sure, that can be done, but we’ll need to stop work on the other three developments we’re working on for you that are due next Monday.”

BP: “No, you can’t stop work on those. I need those next Monday, and this one by next Tuesday.”

Us: “Sure, but we’ll have to exclude the new functionalities that you’d asked for.”

BP: “No, you can’t do that.”

Us:“I’m sorry but if you’re not able to budge on re-prioritising the other work, nor reducing the scope, there’s no way we can hit the timelines you’re asking for, especially when you’re asking for this so late in the game.”

BP: “I’m escalating this. You’ll hear from my manager.”

And so on.

In all fairness though, I have to say that in my experience most managers and senior colleagues (and anyone who has worked in, or closely with, IT) tend to understand that we have to satisfy ourselves with but 24 hours a day to do all we need to do.

These sort of escalations tend to end with “the manager” having a cordial chat with us and agreeing on a workable next step forward, none of which involves us engineering more time into the day.

Establishing a Governance Structure

Having a governance structure tends to minimise “unconstructive” conversations like those above, I think largely because of a mutual trust: the business partner trusts our verdict of whether something is possible or not impossible within a specific time frame,  while we trust that they have thought carefully through their requests and won’t be changing or adding to them unnecessarily.

But the problem with establishing a governance structure is that it, well, needs to be established, which can be incredibly tricky to get going. It’s almost like an negotiating a peace deal, where both sides want the conflict to stop, but are worried what might happen the moment they lay down their arms — will the other side take advantage and strike when they are at their most vulnerable?

I will be the first to admit that it takes a leap of faith going from a world of “if I don’t shout loud enough, and often enough, nothing’s going to get done”, to one where we’re all amicably setting and agreeing on priorities, and where promised delivery deadlines are actually being met.

It also doesn’t help that from a developer’s side, without the benefit of having past projects to tune one’s intuition, accurately estimating project scope or determining deadlines is going to be difficult;  often multiple iterations are necessary before this sort of “accuracy” is achieved. What this means is that early on, chances are good deadlines are going to be missed, which doesn’t help in building trust.

After a missed deadline or two, it’s all too easy to fall back into old patterns and proclaim that the process doesn’t work.

There will also be many, especially those more used to the “free-and-easy” days of yore, who will actively fight the change, citing that it creates too much red tape and jumping through hoops to get things done.

“We need to establish our reporting as soon as possible or we’ll just be flying blind — we can’t afford to go through this process!”

But the thing is, we often can’t afford not to.

When the number of development projects are small, I have to agree that the process, this “bureaucracy”, adds little value. We could simply get on a phone call, or write an e-mail, and agree among ourselves what needs to be done and when. If the requirement changes, no biggie, we simply tweak until its perfect – there’s sufficient slack in the system that will enable us to do just that.

But problems will occur when the number of projects starts to creep up, and more stakeholders are introduced.

The Need for a Tighter Process

The first problem is that due to the higher workload, the slack in the system that allowed for changes in between a development cycle will be gone. This means that changes or additions to the original requirement will likely have to be parked until development time opens up, which could be weeks down the road.

Business partners are not going to like that. “It’s a simple change for God’s sake!”

The thing is, no matter how small a change is, it’s going to be work. Somebody’s got to do it, and that means time out from other projects, which also have agreed timelines. If we focus on that change now, it risks jeopardising the timelines for every other project down the line.

If the change is important enough, then maybe we can take time out from another project and put it into executing the change. But it needs to be agreed by the team owning the other project. Which leads nicely to the second problem.

The second problem is that everyone will have their own agendas, and everyone’s pet project will be “of the highest priority”.

What happens when Team A, B, and C all have “high priority projects” that need to be done by next Monday, and development team only has the capacity to complete one or two? Without a proper process or governance structure, can we guarantee that the project of the highest priority for the business will be one that’s completed?

In the end, more time will be spent explaining to each of the stakeholders why their project was not completed; people will be upset, and the next time they’ll just be sure to shout all the louder, and all the more frequently. More time will be spent on meetings and e-mails, people “ensuring” this and that and never really ensuring anything at all. Estimated delivery dates will be given, but nobody would trust them because they know someone else coming in with an “urgent” request would likely take priority. If it’s “last in, first out”, why should I raise a request early only to be relegated down to the bottom of the delivery pile?

This just struck me as very analogous to the concept of bribery, which I was reminded of on my reading of the book Treasure Islands by Nicholas Shaxson:

Some argue that bribery is ‘efficient’ because it helps people get around bureaucratic obstacles, and get things done. Bribery is efficient in that very narrow sense. But consider whether a system plagued by bribery is efficient and the answer is the exact opposite. [Bribery undermines] the rules, systems, and institutions that promote the public good, and they undermine our faith in those rules.

Despite any short-term drawbacks, there are plenty of longer-term benefits, not least that of supporting stronger surrounding report development structures and a generally healthier culture.

Though setting up the governance structure thus far has been tough, with plenty of push-back and many of our business partners trying to circumvent the process we have established, I think it’s one of the most important things we can, and have ever attempted, to do.

Long vs. Short-term: Doing what needs to be done

There is a huge difference in working with a team that you know will be with you for only a single project and a team that you know will be with you for still many more.

When you’re working with a team that you know will be with you for a long time, you may do what needs to be done to achieve a favourable outcome for this project, but understand the outcome of the sum total of all potential projects to come is just as important, if not more: setting the right precedent and ensuring goodwill among all (as far as is reasonable!) needs just as much attention.

But with the team that you work with for only a single project, you do what needs to be done to achieve the best outcome for this project without too much regard to how that might implicate future interactions with the team. Thinking long-term when you shouldn’t could potentially hurt the outcome of this project.

If you’re only going to be working with them for this one time,  setting a bad precedent or upsetting one or another doesn’t matter too much.

(This post is more a reminder to me than anything else: last year I worked on several one-off projects, during which I was always in this “long-term” mode of thinking. I tried pleasing everybody and making sure I didn’t set poor precedents – “fairly” distributing workload, for example, to people whom I knew couldn’t perform, ultimately hurting the results of these projects.)

On Planning and Project Management

“So how long,” he asks, “do you think you’ll take to complete the project?”

“Two weeks,” I say.

Three weeks later, I’m still two weeks away from completion. What happened?

It’s not like this hasn’t happened before. It’s happened many times before.

Rolf Dobelli says it beautifully in his excellent book The Art of Thinking Clearly:

[W]hy are we not natural-born planners? The first reason: wishful thinking. We want to be successful and achieve everything we take on. Second, we focus too much on the project and overlook outside influences. Unexpected events too often scupper our plans.

The second point is particularly pertinent in my line of work. I have a pretty decent idea of how long a project will take, if all goes well and if I could dedicate 100% of my time to the project.

What I tend to forget is that life often gets in the way. There are plenty of interruptions. Other “important”, “even more urgent” projects pop-up; ad hoc requests that are quick to resolve but that just as quickly add up to significant amounts of time; regular routine work that’s not quite factored in, because they are normally quickly executed, but every once in a while require a long, dragged-out bout of firefighting.

Dobelli has a solution for better planning. He suggests

[shifting] your focus from internal things, such as your own project, to external factors, like similar projects. Look at the base rate and consult the past. If other ventures of the same type lasted three years and devoured $5 million, they will probably apply to your project, too — no matter how carefully you plan.

Which is a great case for keeping a log of project and their time to completion (do NOT trust undocumented recollections of project lengths; like childbirth, our brains tend to underestimate the amount of time and pain we go through).

Also, one more thing I would is that you resist the temptation to think that “this time is different”, and that no matter how confident you are, this time is probably not.

Trust the data.

Shipping like there’s no tomorrow

The concept of shipping as per Seth Godin is a beautiful thing. In essence it relates to the idea of getting something out (i.e. “creative output”; a “product”) without the need to achieve perfection before getting that something out. Ship, get feedback, improve; then ship again, get more feedback, and improve once more. Do this again, and again, and again.

In my role as an analyst, I create a lot of data products that many people use. Things like sales reports; spreadsheet tools and models; and data analyses. Not being one to stick with the status quo, I’m always looking for ways to improve the stuff I create (better interfaces, new metrics,  distribution methods, and more).

In such experimentation (which seeking improvement invariably entails) there’s always a risk that something will go wrong: calculation errors (especially common in complex spreadsheets of which aim is “simple for the user; complex for the developer if need be”); interface usability issues (I tend to work on a large screen that not everybody has or uses); software-compatibility (I use Excel 2013 as I need to know what’s “out there”; which sometimes proves problematic when I use a 2013-only function that 2010-users can’t use); well, the list runs long.

And I’ve had my fair share of failed “improvements” (the latest affecting a couple of very senior management, with my experimentation with Excel Pivot Slicers — very cool stuff by the way — breaking a couple of vital sales reports on their Macs*).

But without having taken these risks, without having shipped, there would be no way that I could be doing as much as I currently do, no way that I could have done as much as I have done. If I had listened to my fears and resisted stepping into the unknown; if I had thought to myself, “let’s not rock the boat, let’s stick with what we know works”, I would be a miserable little automaton working in a far less colourful sales operations environment. And many sales people would still be working with the tools they had in 2013.

Shipping like I do now, though, didn’t come overnight. At the start, I had to push the envelope a little by a little, testing the waters to see just how far I could go. Every little success inspired me further, and built up the all-too-important goodwill that proved useful whenever I messed up;  while every misstep gave me insight into how “open to failure” the environment in which I worked in was (I’ve got very accepting colleagues I must say).

It wasn’t always easy though. Still isn’t. Doubt is a constant companion. I remember once spending three weeks thinking and experimenting with a myriad of ways to capture and report sales targets, a job that I’d been tasked to do.

I was looking for a way that was as intuitive and as “smart” as possible. I must have found at least a hundred ways of how it shouldn’t be done. As the deadline approached, I started feeling like I was being too ambitious, and couldn’t help shaking off the feeling that it had all been wasted time. Maybe it doesn’t have to do all that I wanted it to do.

But I stopped myself. There was still time. “Let’s try,” I told myself, “and let’s ship.” And I did just that.

I shipped an initial “prototype” version, ran it past my boss. It was a rough-and-dirty version, but it showed the concept well. I further developed it, and shipped it past sales management. I showed them screenshots. I did a demo.

What they didn’t realise was that the demo focused on the 10% of what worked; there were plenty of bugs, plenty of things that I hadn’t managed to work out. The actual product was still days away (maybe weeks) from being completed, but I sold them the story, and they bought it. They were excited.

But I certainly wasn’t. I was worried. Worried as hell.

But there was no turning back now. I’d demonstrated what the cool new tool could do (more like what it should do, really). I had no choice. I needed to complete this.

Due over the next few days, I worked like a cow suffering from Bovine Spongiform Encephalopathy (i.e. “mad cow disease”, geddit?) coding and re-coding; testing and re-testing. Eventually some of the most major issues that I had encountered I managed to solve. I shipped again, on the deadline, with 80% of what I had envisioned included.

And boy were they were happy (most of them, anyway). Which made me very happy.

Oh, and the other 20%? Repackaged as “future optimisations”. Which, yes, I eventually did ship too.

* I’m happy to announce that we eventually found a workaround, which appeased the Mac users. The Pivot Slicers remain and prove to be an extremely valuable addition to our reporting arsenal.

Business vs. IT

I felt like a lawyer. The call was in less than 12 hours, and I was busy preparing my case, consolidating evidence and building my story. To be honest, I wasn’t 100% behind the argument I was preparing to put across, but I didn’t really have much of a choice. I had to believe — how could I convince others if I couldn’t even convince myself?

But at the same time, I really wanted to steer clear on the “us vs. them”. Together with allies “from the other side”, we were working hard on framing it from a collaborative angle. It’d do us all no good if our discussion disintegrated into a blame game.

The situation at hand was a classic business vs. IT situation. Business says “we asked for this”, and IT says “no, you didn’t.” Whatever the case, the project deadlines weren’t going to be hit and nobody wanted to be responsible.

The funny thing is, if you asked me, I’d say no one was responsible (or that we all were). It was one of the first times we were doing anything close to what we were doing, and it was almost expected that hiccups like these were going to occur.

Requirements were defined, and IT made good on those requirements. Business was as clear as they could be on those requirements, but apparently not clear enough.  But how could they be? The project was a little too big and too fuzzy to be executed perfectly from the get-go.

Maybe a more iterative approach might have worked better, with all parties agreeing at the start that for a period of say, two or three weeks, we’d all be in a transition phase, where 70% of the requirements were met and the other 30% part of some type of agile-development, exploratory process, where things didn’t need to work perfectly but problems rectified quickly.

True, we could have spent more time thinking through and defining better requirements. But even if we spent an additional year we might not have uncovered requirements buried deep under others, the discovery of which were dependent on the implementation of the others. Would the need for a parachute have come to pass if planes hadn’t yet been invented? Would these later requirements have come to pass if the earlier ones hadn’t been implemented?

Whatever the case, this was an interesting situation to be in.

The Evolutionary Advantage of a Resistance to Change

I was just thinking about organisational change and pondering over how our natural tendency to change is to resist it, when this thought popped up: if resistance to change is so hardwired in our brains, it must serve some purpose — but what?

One of the premises of evolutionary theory is this: if something survives (be it an organism or perhaps even an idea), its survival can be attributed to certain traits or characteristics that help it survive. These traits or characteristics are developed/evolved over time: as those that have these traits survive, and those that don’t die, more and more of the population will have these traits, eventually becoming a “norm”.

Such as a natural tendency to resist change.

At least that’s the theory.

So, let’s make a provocative statement here (inspired by Edward de Bono’s po): Po, a resistance to change isn’t necessarily a bad thing, despite all the bad press it’s getting. In fact, it may be good.

I think that before we get all gung-ho about the next big thing and bashing old ways of doing things, we should think about why things are the way they are, especially if they have been that way for a long time.

Old ways of doing things are, generally speaking, antifragile. They’ve withstood the test of time and have a decent history of working. Good ideas tend to remain good ideas, and the longer they survive the better they tend to be; bad ideas, on the other hand, are discarded as soon as they’re found out (a caveat: those that do manage to survive for long, however, tend to be the most dangerous — bad things are made worse when they’re not known to be bad. Think insidious. Think CFCs.)

So if you have an old way of doing things, it may not be the best way, but it’s likely a way that generates decent results, enough for it to have lasted as long as it has. The moment a better way of doing things is found and tested to work, the old way is discarded. But until then, the old way is the best way.

If people weren’t resistant to change, on the other hand, good ideas wouldn’t have the time to spread. We’d be flitting from one idea to the next, discarding great ones and embracing bad ones in equal measure.

So a resistance to change isn’t all that bad. It’s the way things should be. The incumbent has earned its right on the throne, and the onus should always be on the challenger to prove its worth.

So it does worry me if people rush into new ways of doing things without having redundant systems in place, just in case. I mean, let’s not be too hasty in burning bridges.

If there’s going to be a process change, have the old process remain in place until the new process is proven stable. Depending on what the process is, a few iterations (or days, or months) is most certainly needed. Give it time to prove its antifragility.

When automobiles were first introduced, horse-drawn carriages didn’t disappear overnight. Concurrency. Then obsolescence.

Resist, consider, then change. Carefully.