Why 3 Point Estimates Create False Optimism - Part 2 (Guest Blog)

Thanks to Glen Alleman for this guest blog

The use of 3 point estimates is fraught with statistical integrity problems. Don't elicit estimates in this way.

While many will have anecdotal evidence of this working for them personally on small self contained projects, the issue of estimating impacts enterprise, multiple supplier, and integrated project teams (IPTs).

Summary of the Three Point Estimate Approach to Cost and Schedule

• The estimates are biased values
• The estimates are not anchored to the underlying probability distributes
• The estimates are not correlated with the other estimates in the project network or cost structure model

The Details

This approach starts with two primary sources. Effective Risk Management, Dr. Edmund Conrow, AIAA, 2003 and "Judgment Under Uncertainty: heuristics and Biases," Tversky and Kahnemanm Science, Volume 185, 27 September 1974.

People often rely on a limited number of heuristic principles that reduce complex tasks of assessing probabilities and predicting values to simpler judgmental operations. These heuristics can lead to biased assessments of probability. Three such heuristics, first discussed by Yversky and Kahneman, include adjustment and anchoring, availability, and representativeness.

The broader discussion of these biases can be found in the full length discussion in Judgment Under Uncertainty: Heuristics and Biases, Cambridge University Press 1982. This is a primary source work. Conrow and others have derived the impacts from this material. Primary impacts can be found in Complex estimating problems in software development, large construction, oil & gas capacity estimating (field production capacities), and economics.

As stated by Tversky and Kaheman: "In many situation people make estimates by starting from the initial value that is adjusted to yield the final answer." The judgment heuristic is called "adjustments. Adjustments are typically insufficient - different starting points yield different estimates which are biases toward the initial values (hence the term anchoring).

The consequences of adjustment and anchoring lead to an underestimation bias of potential minimum and maximum values associated with the likelihood of an event, a duration, or the probability distribution representing these activities.

So Now the 3 Point Estimate Problem - Part 2

Many of the probabilistic questions (what is the likely duration of the task?) with which people are concerned belong to one of the following types: 

• What is the probability of an object A belonging to a class B? 
• What is the probability that process B will generate event A? 

In these cases, people often rely on representativeness in which the probabilities are evaluated by the degrees to which A is representative of B, that is, by the degree to which A resembles B.

This is what happens when you ask a developer or planned to define the Most Likely duration for work and then ask what is the upper and lower limits of that duration. A Naval Research study and studies in the oil & gas field production estimating domain  have clearly shown that the order in which you ask those question results in statistically significant difference in the estimated values.

But There is More

The heuristic above is called representativeness and it affects risk related decisions (duration estimates are actually risk estimates - the risk of completing on or before a specific date). This risks include:

1. Insensitivity to prior probability - the Bayesian statistical impacts spoken about by Kailash in Bayes Theorem for Project Managers.
2. Insensitivity to sample size - this is the classic anecdotal evidence of "it works for me on my sample projects, so it must work in general."
3. Misconceptions of change - the drawing of a triangle or any arbitrary curve, placing to end points and marking the peak of the curve and drawing conclusions in the absence of any confirm that this pseudo-probability distribution has any connection what so ever with reality. Or worse, reverse engineering a probability distribution curve to fit the data for one project and then generalizing that process to all projects in the universe. This latter is how the Beta curve, with fixed symmetric standard deviations resulted in the PERT formula. PERT can have up to 27% unfavorable bias in the presence of coupling between tasks.
4. Insensitivity to predictability - again a Bayesian problem.
5. The illusion of validity - no calibrated variances.
6. Misconceptions of regression.

It's the first of these heuristics that is the source of the exclusion of the 3-point process as practiced by many - asking the subject matter expert for the three values.

When faced by ambiguity or uncertain information, people have a tendency to interpret information that confirms their beliefs; with new data they tend to accept information that confirms their beliefs but to question new information that conflicts with them.

Improving Risk Communication, National Research Council, National Academy Press, Washington DC, 1989.

Conclusion

Making the 3 point estimate process the basis of project cost and schedule estimating is very sporty business. For personal projects, projects where you are the personal lead, have a small group of friends working the project, or the project is essentially self contained with your small group of friends, the risk is lower.

If your project is being implemented in a larger context, by essentially strangers - contractors, teams beyond your direct experience and control (other staff teams, development teams from other locations, subcontractors, 3rd parties, COTS providers) - you're taking on risk and may not know it, be able to quantify it, or even characterize this risk profile.

This is a primary source of project failure - there is no credible source of cost and schedule estimates when you simply ask some "what is the most likely, upper, and lower limits of cost or duration." You've planted the seeds of late and over budget and may not even know it.