Expert Opinion is Overrated, or The Power of Simple Formulas

If you haven’t yet read Thinking, Fast and Slow by Daniel Kahneman, I cannot encourage you strongly enough to do so. In this book, Prof. Kahneman – a winner of the Nobel Prize in Economic Sciences – summarizes his lifetime of research into the workings of the human mind, our decision-making process and the fallacies we are susceptible to.

There are a lot of eye-openers in the book but a few stand out. One is found in the chapter called “Intuitions vs. Formulas”. The lesson in this chapter is extremely provocative:

Simple formulas often outperform expert judgment.

What this means is that a simple formula to predict future performance may be more reliable than expert opinion.

The key example provided in the chapter is Kahneman’s own story of creating an interview system for the Israeli army. Prior to Kahneman, the evaluation of a recruit’s combat fitness was chiefly made by an expert – the interviewer who tried to forecast how well the recruit would do.

It turned out that the expert judgment was practically useless in predicting the future success of recruits. At age 21, Kahneman was tasked to create a better approach. Shortly before, Kahneman read a book by the psychologist Paul Meehl about statistical prediction, and he decided to put Meehl’s theories into practice.

Objective Inputs + Standard Formulas

The new interview system included separate scoring of several personality traits in each recruit that would appear relevant to his performance in a combat unit – traits like responsibility, sociability and “masculine pride”. For scoring each personality trait, Kahneman came up with a series of standardized questions. The candidate’s final score was computed as a simple sum of individual scores.

The evaluations held a few months after the new system was installed showed that it was a substantial improvement over the old one. A simple sum of the six scores predicted the soldier’s performance much more accurately that the previous, expert-based method.

Now, you may be inclined to think that the improvement was due to Kahneman systematizing the interview process by making the interviewers score specific personality traits in each recruit. While there is undoubtedly value in this, the crucial – crucial! – factor was what questions were used for scoring these traits.

Kahneman writes: “I… composed, for each trait, a series of factual questions about the individual’s life before their enlistment.” For example, questions for scoring responsibility included how many different jobs the recruit had held or how punctual he had been in his work and studies.

Contrast this with a typical job interview where the interviewer would ask questions like “Do you consider yourself responsible?”, “Give me an example where you showed yourself as a responsible person”, or “Imagine you found yourself in such and such situation. What would you do?”

Kahneman’s system included none of this. Instead, the scoring in his system was based on questions about objective facts of the person’s life that are indicative of the personality trait in question.

In particular, here is what Kahneman has to say:

I concluded that the then current interview had failed at least in part because it allowed the interviewers to do that they found most interesting, which was learn about the dynamics of the interviewee’s mental life.

Not Just for the Army

The superiority of simple formulas over expert judgment has been confirmed in a multitude of areas besides army recruitment. Here are a few other examples – some are given in the book and some are my observations:

Predicting the future value of wines. Bottles of fine Bordeaux wine filled only 12 months apart can differ in value by a factor of 10 or more. A statistical formula developed by the Princeton economist Orley Ashenfelter predicts the future price of wine by just three features of the weather of the bottling year. The Ashenfelter’s formula provides accurate price forecasts years and even decades into the future and is much more accurate than expert opinion.

Marital stability. Robyn Dawes showed that marital stability is well predicted by a simple formula:

Stability = frequency of lovemaking – frequency of quarrels

Basically, you don’t want your stability number to be in the negative.

Stock market investing. It is a well-established fact that index investing, or formula-based investing into a basket of stocks representative of the market is a whole, outperforms a vast majority of fund managers over the long run. “A low-cost index fund is the most sensible equity investment for the great majority of investors,” agrees Warren Buffett, arguably the best investor history has known.

Evaluating physical distress of newborns. A rating scale developed by anesthesiologist Virginia Apgar ranks newborns across five variables (heart rate, respiration, reflex, muscle tone, and color) on a scale of 0 to 2 one minute after they are born. The scale has proved to be much more accurate than clinical judgement in determining whether the baby is in distress and needs medical help. Apgar score has saved lives of hundreds of thousands of infants since its introduction in 1953.

Making the Formulas Work for You

In the end of the “Intuitions vs. Formulas” chapter, Kahneman urges the reader to put the formulas to work in their life and provides a detailed description of his formula-based method for assessing job candidates.

While this method requires “relatively little effort,” Kahneman warns that it also requires “substantial discipline.” This has been exactly my experience in implementing the interview procedures in Kahneman’s spirit in my company, and here is why.

As I mentioned before, the key to the success of Kahneman’s formula-based approach is the objective, factual inputs – i.e. questions used to evaluate each of the job candidate’s 5 or 6 personality traits. When devising the questions, I had to constantly remind myself to avoid questions dealing with the dynamics of the candidate’s “mental life” and use fact-based questions instead.

For example, if you were to evaluate the candidate’s ability to operate in unknown, you may be tempted to ask, “How did you deal with changes at work?”, and that would be a wrong question. Instead, one better – factual – question used to assess this trait would be, “Have you traveled a lot?”

My hiring has been very successful so far, and I attribute much of the success to the usage of a formula instead of subjective judgement. Try putting simple formulas to work for you, and you will be surprised with the results.

Leave a Reply

Your email address will not be published. Required fields are marked *