Daniel Kahneman breaks down why this happens and how to respond in Thinking Fast and Slow (chapter 23). We are all prone to "The Planning Fallacy" and this is because we lack "The Outside View."
He writes: "The treatment for the planning fallacy has now acquired a technical name, REFERENCE CLASS FORECASTING" and "the renowned Danish planning expert Bent Flyvbjerg... has applied it to transportation projects in several countries."
The process has 3 steps which boils down to getting data on a representative group and setting a baseline, and then evaluating how well your case compares to the baseline so you can adjust.
At a personal management and project management level, we will always have practical difficulty dedicating any time to gathering baseline data if we don't already have it. I've found that, to your point Wes, we are best off taking our initial assumption and padding it with a substantial contingency that grows with the uncertainty and complexity of the task.
One could argue that „there is nothing new under the sun“, and hence that this kind of baselining is always possible. I think though that there are cases - typically new and shiny initiatives - where both your *own* and, more importantly, your stakeholder‘s expectations are extremely high („shiny factor“) and where the „Planning Fallacy“ will hit you even harder. Finding a realistic baseline for that kind of bootstrapping can be quite challenging, if not impossible.
The best strategy I’ve found is to a) assume it will take longer than you think, b) be quite stubborn about signaling to your external world that „it will take whatever time it takes“ as you figure out the next best step, and c) work relentlessly but humbly towards that next step, reducing complexity and thus uncertainty along the way.
I make a living working in ambiguous and high velocity situations like what you describe, and your recommendation is spot on. I've also learned that there are often more similar cases out there than apparent novelty might suggest, so when it's possible to slow down enough to identify some comparable cases, if not datasets - and solicit an outside view - it can expose bias that would otherwise go overlooked.
I love the reminder about the frustration factor. Generally when approaching something new we are optimistic and naive. We can’t help but be these things so remembering and maybe simply multiplying the estimate by 3 can help with the “less positive” self talk that follows. Thanks!
Everything takes longer than we think (statistically speaking, of course; there always are outliers).
There are two main reasons.
1. There are only as many things that may happen which would make the task go *faster* than we expect. While there are almost an infinite number of things that can make the thing go *slower*.
The game is stacked against us.
Even if we take into account a buffer for some unexpected stuff, there's still typically not enough. We'd intuitively think that it's equally likely that something good will happen as it is that something bad will.
In reality, because there are many more "negative" events, it's much more likely that a random event will slow us down and not speed us up.
2. We notoriously think of our time as if we were productive 100% of the time (or close).
In reality, we waste a lot of time. All the context switches, breaks, toilet breaks, coffee breaks, random chats with colleagues, a call from an unknown number, an ambulance making noise outside, etc.
Individually, each of these can be negligible, but in total, they stack up to a significant time.
If we consider so-called value-adding work, it's even worse. For each new feature we create, there is a bug fix, two meetings, a lunch break, and a 2-minute-long internal fight to overcome procrastination.
In a couple of organizations, I measured how much teams were spending on value-adding work. They were consistently in a range of 30%-40%.
Surprisingly, if they multiplied their estimates by a factor of 3, they might have been pretty accurate.
That's why inferring the time estimates from the historical data is so powerful. If I finish 3 features on average every week, it's likely that I will land in a similar neighborhood next week, too. Because my historical data *includes* my lunches, going to the loo, random chats, and all the other stuff, too.
The magic trick is decoupling estimates with time assessment per task, and counting how many things have been completed over a longer period of time.
Great point. To your point above, we don't consider the use cases when we set the baseline expectation.Its more of a default response to set the expectation that if it worked for this case, it will work for another. I do tend to add some buffer to allow for issues - project management principle =)
How would you handle a situation where you give an estimate of how long it’ll take to complete a task, and your manager/stakeholder asks why does it take that long? Why can’t you be more efficient?
When we are young and idealistic, and work in high-efficiency, high-demand environments, we have strong expectations of time, both expecting and wanting everything to happen quickly. But as we get older we just realize everything happens slower than expected or desired. Progress in career. Parenting. Finding meaning. Fitness. Friendships. Marriage. Wisdom. Nothing good, rewarding, and worthy of pursuit happens in a day. So we may as well learn to adjust our expectations, expect interruptions, and enjoy the journey along the way.
I once had a five year plan and was surprised to check it and realize I had accomplished the work in three. So I think things can take longer than you think or shorter - it’s important to track. This way you can gain additional insights into why.
Daniel Kahneman breaks down why this happens and how to respond in Thinking Fast and Slow (chapter 23). We are all prone to "The Planning Fallacy" and this is because we lack "The Outside View."
He writes: "The treatment for the planning fallacy has now acquired a technical name, REFERENCE CLASS FORECASTING" and "the renowned Danish planning expert Bent Flyvbjerg... has applied it to transportation projects in several countries."
The process has 3 steps which boils down to getting data on a representative group and setting a baseline, and then evaluating how well your case compares to the baseline so you can adjust.
At a personal management and project management level, we will always have practical difficulty dedicating any time to gathering baseline data if we don't already have it. I've found that, to your point Wes, we are best off taking our initial assumption and padding it with a substantial contingency that grows with the uncertainty and complexity of the task.
One could argue that „there is nothing new under the sun“, and hence that this kind of baselining is always possible. I think though that there are cases - typically new and shiny initiatives - where both your *own* and, more importantly, your stakeholder‘s expectations are extremely high („shiny factor“) and where the „Planning Fallacy“ will hit you even harder. Finding a realistic baseline for that kind of bootstrapping can be quite challenging, if not impossible.
The best strategy I’ve found is to a) assume it will take longer than you think, b) be quite stubborn about signaling to your external world that „it will take whatever time it takes“ as you figure out the next best step, and c) work relentlessly but humbly towards that next step, reducing complexity and thus uncertainty along the way.
I make a living working in ambiguous and high velocity situations like what you describe, and your recommendation is spot on. I've also learned that there are often more similar cases out there than apparent novelty might suggest, so when it's possible to slow down enough to identify some comparable cases, if not datasets - and solicit an outside view - it can expose bias that would otherwise go overlooked.
I love the reminder about the frustration factor. Generally when approaching something new we are optimistic and naive. We can’t help but be these things so remembering and maybe simply multiplying the estimate by 3 can help with the “less positive” self talk that follows. Thanks!
Everything takes longer than we think (statistically speaking, of course; there always are outliers).
There are two main reasons.
1. There are only as many things that may happen which would make the task go *faster* than we expect. While there are almost an infinite number of things that can make the thing go *slower*.
The game is stacked against us.
Even if we take into account a buffer for some unexpected stuff, there's still typically not enough. We'd intuitively think that it's equally likely that something good will happen as it is that something bad will.
In reality, because there are many more "negative" events, it's much more likely that a random event will slow us down and not speed us up.
2. We notoriously think of our time as if we were productive 100% of the time (or close).
In reality, we waste a lot of time. All the context switches, breaks, toilet breaks, coffee breaks, random chats with colleagues, a call from an unknown number, an ambulance making noise outside, etc.
Individually, each of these can be negligible, but in total, they stack up to a significant time.
If we consider so-called value-adding work, it's even worse. For each new feature we create, there is a bug fix, two meetings, a lunch break, and a 2-minute-long internal fight to overcome procrastination.
In a couple of organizations, I measured how much teams were spending on value-adding work. They were consistently in a range of 30%-40%.
Surprisingly, if they multiplied their estimates by a factor of 3, they might have been pretty accurate.
That's why inferring the time estimates from the historical data is so powerful. If I finish 3 features on average every week, it's likely that I will land in a similar neighborhood next week, too. Because my historical data *includes* my lunches, going to the loo, random chats, and all the other stuff, too.
The magic trick is decoupling estimates with time assessment per task, and counting how many things have been completed over a longer period of time.
Also: Reality Has a Surprising Amount of Detail [1] (A classic!).
[1]: Salvatier, John. “Reality Has a Surprising Amount of Detail.” John Salvatier (blog), May 13, 2017. http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail.
Include an “auto add buffer” of time for managing stakeholders?
Great point. To your point above, we don't consider the use cases when we set the baseline expectation.Its more of a default response to set the expectation that if it worked for this case, it will work for another. I do tend to add some buffer to allow for issues - project management principle =)
How would you handle a situation where you give an estimate of how long it’ll take to complete a task, and your manager/stakeholder asks why does it take that long? Why can’t you be more efficient?
Agreed. Building a quality product takes so much time, effort, sweat, and tears. So much work behind the scenes that is never seen.
"We do this not because it is easy. but because we thought it would be easy."
Fascinating read! I’m Harrison, an ex fine dining line cook. My stack "The Secret Ingredient" adapts hit restaurant recipes for easy home cooking.
check us out:
https://thesecretingredient.substack.com
When we are young and idealistic, and work in high-efficiency, high-demand environments, we have strong expectations of time, both expecting and wanting everything to happen quickly. But as we get older we just realize everything happens slower than expected or desired. Progress in career. Parenting. Finding meaning. Fitness. Friendships. Marriage. Wisdom. Nothing good, rewarding, and worthy of pursuit happens in a day. So we may as well learn to adjust our expectations, expect interruptions, and enjoy the journey along the way.
I once had a five year plan and was surprised to check it and realize I had accomplished the work in three. So I think things can take longer than you think or shorter - it’s important to track. This way you can gain additional insights into why.