One of the more recent trends I’ve noticed is organisations having trouble with the Agile Convention (and Scrum Rule) of:

“The Product Backlog is estimated by The Team Who Is Going To Implement it”.

In fact some folks have so much trouble with the concept that they deny that it’s even remotely possible.

Digging a little deeper I’ve found that in many organisations there is (to my mind at least) an enormous gap between defining a project and executing a project; and that these two “phases” are typically conducted by two entirely different sets of people.

And yes they’re referring to these endeavours as Agile Projects.

In actuality, that’s basically Waterfall, and as such it’s missing out on a lot of the point.

There are two fundamental issues here:
1. How these Estimates are being used
2. How Knowledge in general is being transferred

Let’s talk about Knowledge Transfer first.

Tacit vs Explicit Knowledge

Waterfall is based around the concept of Explicit Knowledge. It operates on the fundamental assumption that we can (always) transfer knowledge between individuals through the medium of explicit information – i.e. writing it down.

It is this belief in the universal applicability of explicit knowledge that allows Waterfall to function. What is effectively (supposed to be) happening in Waterfall is progressive translation of the same information from one explicit knowledge format to the next. Much of the focus of management on Waterfall Projects is verification that this translation has occurred without error or omission.

Agile however is based around the belief that Explicit Information has its limitations and that one of the primary causes of failure in Waterfall projects is that progressive translation instead becomes progressive interpretation (Think of the children’s game telephone) – and thus the introduction of mistakes is inevitable.

As such, Agile leverages more heavily on Tacit Information i.e: the stuff in people’s heads. (Face to Face communication anybody?)

“Effective transfer of tacit knowledge generally requires extensive personal contact and regular interaction” [1]

What this means in practice is that Agile relies heavily on co-creation not interpretation; fundamentally every artefact in Agile serves as a tacit reminder of intent and purpose for those people who took part in that artefact’s creation. Eliminating or at the very least reducing the “telephone effect”.

And this is also why there is such a fanatical focus on producing and evaluating “Working Software” – because not only is it clearly possible for communication to break down at any step in the process of creating software; the arguably most important part to verify is whether or not the software solves its intended problem – and it’s those two “points” have the most layers of potential “re interpretation” between them.

If you’re unaware of the distinction between Tacit and Explicit information, then this will simply look like a waste of time to you[2] and instead you’ll think nothing of writing a 4 inch high stack of User Story Cards all by yourself.

In Waterfall, we initially have no choice but to interpret the words that come to us from the previous stage but then we are able to do whatever suits us as long as we can reasonably defend our interpretation.

In Agile however, we strive to intentionally interact and co-create value such that our words, should we write them down, are not subject to interpretation, but rather trigger memories of rich interactions.[3]

But of course, all of this can only happen if it was the same group of people.

Guessing about Grooming

So already you can begin to see the problem with separating the “Estimation” people from the “Implementation” people.

In these situations “Grooming” and “Story Writing” tends to drag on forever as:
1. We attempt to transfer all our knowledge onto paper (or into JIRA) – which is slow and lossy.
2. We have to guess[4] at when we’re done with Task #1 [5]

The results of this can either be “bad” or “worse”.

Bad

Bad is where the “Estimation People” spent sufficient time with the “Definition People” that they do[6] in fact have a solid tacit understanding of what needs to be done.[7]

Worse

In the worse scenario – the “Estimation People” are just as much in the dark as the ultimate “Implementation People” – and they’re also just interpreting whatever they were given.

The Real Point of Estimation in Agile

Fundamentally there are two main reasons why you might want to estimate a piece of work

1. To aid in planning and prioritisation
2. To clarify understanding

With point (2) being a fundamental requirement for Point (1).

Estimates can work as a useful “out of band” abstraction for a piece of work that when incredibly divergent – can act as a signal that understanding is not uniform[8]

i.e: if our estimates are greatly diverse – then somebody, maybe everybody’s understanding is wrong.[9]

Additionally, getting people to externalise a number aids in getting them to engage with the problem in a way that simply asking “Do we all understand this” – never will.

My Guesses become Your Shackles

In both cases above however, the clarification step is skipped and the externalisation effect ceases to have any merit.

What we end up with instead is a set of estimates that reflect a mixture of typically two things:

1. What the stakeholders want the software to cost
2. What opinion the “Estimation People” have of the “Implementation People”

What do we want this to cost?

I have previously covered the issue of estimating to fit a commitment, suffice to say that separating your Estimators from your Implementors makes this task easier and the eventuality of it happening more likely[10], as the Estimators only have to make the client happy and know that they’ll never be on the hook for delivering to what they may well know is a ludicrous timetable.

The sad part about all of this, is that fixing the cost for Software Development Projects is a problem that Agile is pretty good at solving – as long as people let go of their old habits, like creating estimates for other people.

What do I think about you?

And lastly, if we’re having to estimate how long somebody else is going to take to do something, then we have to have some kind of impression about their ability. And this typically falls into two categories:

1. I am competent, and they should take the same amount of time as me
2. I am really competent, there is no way in which they will be able to get this done as quickly as me.

So this goes wrong in all sorts of ways, assuming that the Estimators are actually the more senior people, then choosing Option 1 usually ends up in underestimation, especially if the Estimators actually over estimate their own brilliance (which they are of course incentivised to do, as their own abilities are never actually going to be tested here)

But the downside of Option 2 is rarely considered.

Estimates become targets or commitments

The unspoken assumption behind separating out Estimation from Implementation is that the Estimate actually becomes a statement of “How long something should take” – if the people doing the work are:
1. Competent
2. Working Hard Enough

Thus doing the work within the prescribed amount of time becomes a new meta goal (which in certain pathological circumstances usurps the original goal of solving a Real World™ Problem)

This means that upfront, estimation puts a cap on our productivity and a damper on our creativity. The goal now becomes to expend the time, not to solve the problem, and certainly not to do anything clever or creative in order to solve the problem! (imagine taking way longer for one part of work because you’re building a capability which makes the rest of the work take 10th of the time)

And of course quality is the traditional sacrificial lamb in these scenarios:

“Well you didn’t ‘estimate’ me enough time to do this to any reasonable level of quality”

In Summation

The underlying belief here seems to be that for many organisations “A Project” exists as an entirely distinct entity from the people who will eventually build it. Much like a recipe in a cookbook. [11]

There are four problems with this however:

1. Your Project is not a recipe in a cookbook [12]
2. The clock is still running during waiting time [13]
3. Time and Effort is wasted in both the project definition and at the uptake of execution – as the project definers have to spend additional effort in fleshing out the detail they assume the (unknown) implementers are going to require;
4. Explicit Knowledge is imperfect. And mistakes are made not only in execution, but also simply in the understanding.

None of these four things are good. Put them all together and they add up to:

1. Projects that cost too much [14]
2. Projects that take too long – especially from the Customer’s Perspective
3. Projects delivering the wrong thing™
4. Generally everybody focussed on the wrong thing [15]

An Agile Project is just as much about the people as it is about the plan

Agile ways of working solve this problem – not through process, but rather through understanding that people are, well, human.

• Writing notes to your future self is more effective than writing them to somebody else
• Estimating how long you’re going to take to do something is always going to be more accurate and motivating than figuring out how long somebody else might take to do it

I’m beginning to think that the concept of #NoEstimates may not go far enough some times. Maybe we should probably stop using the word “Estimate” altogether, at least for a while and start using slightly more precise terms such as “How Long This Would Take If I Was to do it”, “The Maximum Amount of money I’m willing to pay for this functionality” and “The amount of time I’d like this to take which is based off no information apart from how I feel about the situation”

But if we can’t do that, then at least we should start socialising the concept that Agile Projects are made out of people and not paper.[16]