One of the more recent trends I’ve noticed is organisations having trouble with the Agile Convention (and Scrum Rule) of:
“The Product Backlog is estimated by The Team Who Is Going To Implement it”.
In fact some folks have so much trouble with the concept that they deny that it’s even remotely possible.
Digging a little deeper I’ve found that in many organisations there is (to my mind at least) an enormous gap between defining a project and executing a project; and that these two “phases” are typically conducted by two entirely different sets of people.
And yes they’re referring to these endeavours as Agile Projects.
In actuality, that’s basically Waterfall, and as such it’s missing out on a lot of the point.
There are two fundamental issues here:
1. How these Estimates are being used
2. How Knowledge in general is being transferred
Let’s talk about Knowledge Transfer first.
Tacit vs Explicit Knowledge
Waterfall is based around the concept of Explicit Knowledge. It operates on the fundamental assumption that we can (always) transfer knowledge between individuals through the medium of explicit information – i.e. writing it down.
It is this belief in the universal applicability of explicit knowledge that allows Waterfall to function. What is effectively (supposed to be) happening in Waterfall is progressive translation of the same information from one explicit knowledge format to the next. Much of the focus of management on Waterfall Projects is verification that this translation has occurred without error or omission.
Agile however is based around the belief that Explicit Information has its limitations and that one of the primary causes of failure in Waterfall projects is that progressive translation instead becomes progressive interpretation (Think of the children’s game telephone) – and thus the introduction of mistakes is inevitable.
As such, Agile leverages more heavily on Tacit Information i.e: the stuff in people’s heads. (Face to Face communication anybody?)
“Effective transfer of tacit knowledge generally requires extensive personal contact and regular interaction” 
What this means in practice is that Agile relies heavily on co-creation not interpretation; fundamentally every artefact in Agile serves as a tacit reminder of intent and purpose for those people who took part in that artefact’s creation. Eliminating or at the very least reducing the “telephone effect”.
And this is also why there is such a fanatical focus on producing and evaluating “Working Software” – because not only is it clearly possible for communication to break down at any step in the process of creating software; the arguably most important part to verify is whether or not the software solves its intended problem – and it’s those two “points” have the most layers of potential “re interpretation” between them.
If you’re unaware of the distinction between Tacit and Explicit information, then this will simply look like a waste of time to you and instead you’ll think nothing of writing a 4 inch high stack of User Story Cards all by yourself.
In Waterfall, we initially have no choice but to interpret the words that come to us from the previous stage but then we are able to do whatever suits us as long as we can reasonably defend our interpretation.
In Agile however, we strive to intentionally interact and co-create value such that our words, should we write them down, are not subject to interpretation, but rather trigger memories of rich interactions.
But of course, all of this can only happen if it was the same group of people.
Guessing about Grooming
So already you can begin to see the problem with separating the “Estimation” people from the “Implementation” people.
In these situations “Grooming” and “Story Writing” tends to drag on forever as:
1. We attempt to transfer all our knowledge onto paper (or into JIRA) – which is slow and lossy.
2. We have to guess at when we’re done with Task #1 
The results of this can either be “bad” or “worse”.
Bad is where the “Estimation People” spent sufficient time with the “Definition People” that they do in fact have a solid tacit understanding of what needs to be done.
In the worse scenario – the “Estimation People” are just as much in the dark as the ultimate “Implementation People” – and they’re also just interpreting whatever they were given.
The Real Point of Estimation in Agile
Fundamentally there are two main reasons why you might want to estimate a piece of work
- To aid in planning and prioritisation
- To clarify understanding
With point (2) being a fundamental requirement for Point (1).
Estimates can work as a useful “out of band” abstraction for a piece of work that when incredibly divergent – can act as a signal that understanding is not uniform
i.e: if our estimates are greatly diverse – then somebody, maybe everybody’s understanding is wrong.
Additionally, getting people to externalise a number aids in getting them to engage with the problem in a way that simply asking “Do we all understand this” – never will.
My Guesses become Your Shackles
In both cases above however, the clarification step is skipped and the externalisation effect ceases to have any merit.
What we end up with instead is a set of estimates that reflect a mixture of typically two things:
- What the stakeholders want the software to cost
- What opinion the “Estimation People” have of the “Implementation People”
What do we want this to cost?
I have previously covered the issue of estimating to fit a commitment, suffice to say that separating your Estimators from your Implementors makes this task easier and the eventuality of it happening more likely, as the Estimators only have to make the client happy and know that they’ll never be on the hook for delivering to what they may well know is a ludicrous timetable.
The sad part about all of this, is that fixing the cost for Software Development Projects is a problem that Agile is pretty good at solving – as long as people let go of their old habits, like creating estimates for other people.
What do I think about you?
And lastly, if we’re having to estimate how long somebody else is going to take to do something, then we have to have some kind of impression about their ability. And this typically falls into two categories:
- I am competent, and they should take the same amount of time as me
- I am really competent, there is no way in which they will be able to get this done as quickly as me.
So this goes wrong in all sorts of ways, assuming that the Estimators are actually the more senior people, then choosing Option 1 usually ends up in underestimation, especially if the Estimators actually over estimate their own brilliance (which they are of course incentivised to do, as their own abilities are never actually going to be tested here)
But the downside of Option 2 is rarely considered.
Estimates become targets or commitments
The unspoken assumption behind separating out Estimation from Implementation is that the Estimate actually becomes a statement of “How long something should take” – if the people doing the work are:
2. Working Hard Enough
Thus doing the work within the prescribed amount of time becomes a new meta goal (which in certain pathological circumstances usurps the original goal of solving a Real World™ Problem)
This means that upfront, estimation puts a cap on our productivity and a damper on our creativity. The goal now becomes to expend the time, not to solve the problem, and certainly not to do anything clever or creative in order to solve the problem! (imagine taking way longer for one part of work because you’re building a capability which makes the rest of the work take 10th of the time)
And of course quality is the traditional sacrificial lamb in these scenarios:
“Well you didn’t ‘estimate’ me enough time to do this to any reasonable level of quality”
The underlying belief here seems to be that for many organisations “A Project” exists as an entirely distinct entity from the people who will eventually build it. Much like a recipe in a cookbook. 
There are four problems with this however:
- Your Project is not a recipe in a cookbook 
- The clock is still running during waiting time 
- Time and Effort is wasted in both the project definition and at the uptake of execution – as the project definers have to spend additional effort in fleshing out the detail they assume the (unknown) implementers are going to require;
- Explicit Knowledge is imperfect. And mistakes are made not only in execution, but also simply in the understanding.
None of these four things are good. Put them all together and they add up to:
- Projects that cost too much 
- Projects that take too long – especially from the Customer’s Perspective
- Projects delivering the wrong thing™
- Generally everybody focussed on the wrong thing 
An Agile Project is just as much about the people as it is about the plan
Agile ways of working solve this problem – not through process, but rather through understanding that people are, well, human.
- Writing notes to your future self is more effective than writing them to somebody else
- Estimating how long you’re going to take to do something is always going to be more accurate and motivating than figuring out how long somebody else might take to do it
I’m beginning to think that the concept of #NoEstimates may not go far enough some times. Maybe we should probably stop using the word “Estimate” altogether, at least for a while and start using slightly more precise terms such as “How Long This Would Take If I Was to do it”, “The Maximum Amount of money I’m willing to pay for this functionality” and “The amount of time I’d like this to take which is based off no information apart from how I feel about the situation”
But if we can’t do that, then at least we should start socialising the concept that Agile Projects are made out of people and not paper.
- Goffin, K. & Koners, U. (2011). Tacit Knowledge, Lessons Learnt, and New Product Development. ↩
- Never time to do it right, but always time to do it over. ↩
- Another reason to work in short cycles, the human memory fades. ↩
- Or worse, somebody else has a made a universal guess on our behalf – and prescribed what a “Good User Story” looks like in the abstract. This is great for arse covering, but tends to be horrible for getting great stuff built. ↩
- And because we tend to attribute failure to the (wo)man and not the method, the next time around we simply spend longer and write more. And thus miss the point yet again, but this time by a wider margin. ↩
- Or at least think that they do. This is probably where it all starts going wrong. ↩
- I will leave it as an exercise for the reader as to why then they do not then just go ahead and do the work themselves. ↩
- This is why it’s so important that (in order to be useful) Story Points are regarded as Team Estimates and not Individual, or “skill class” estimates. ↩
- If the estimates are all the same however, this does not necessarily follow that understanding is uniform however – beyond the size of the work. If the work unit is small enough, this should hopefully not matter too much. ↩
- People like to be liked. And they also like to discount the future. In the vacuum of their being any other consequence, most people, if put in this situation, will come up with a number that makes the requester happy in that moment. ↩
- Often coupled with the associated belief that the recipe is without fault, and that it’s only the cook who can fail to follow the instructions correctly. ↩
- For a start, if it’s a half decent cookbook then somebody will have tried all the recipes first. Hell, I can almost guarantee that they iterated their way to creating the recipes. Nobody has tried your project first, at least not to completion; and if they have – save yourself a whole lot of time and risk and just buy it off them. ↩
- From a accountant’s perspective they may however be happy that nobody is billing towards the project; however the users, who presumably had an actual need, are losing money – either real or opportunity terms.
There is a secondary issue here as well, in that the longer we wait to deliver requested functionality, the more likely it is to be out of date by the time it’s delivered.
To extend the analogy, what we are effectively doing with our projects is thinking “I’m hungry, what’s for dinner?” and then waiting for somebody else to write and publish a cookbook. ↩
- Not deliberately however. I think the most common reasoning is that by separating out the planning from the execution, that money can be saved on finding the cheapest possible executors. See any of my more recent sessions on Agile Contracting for a discussion on this. ↩
- Meeting the estimate, not solving the problem. ↩
- Which given sheer amount of paper stuck to the walls on the average Agile Project seems mildly ironic 😉 ↩