16 January 2012

We are Not as Smart as We'd Like to Think

Writing at VoxEu.org Victor Ginsburgh, of the Université Libre de Bruxelles, notes that experts aren't so expert in many situations:
A paper by Fritz et al (2012) published last week in the Proceedings of the National Academy of Sciences shows that professional musicians are unable to distinguish between the tonal superiority of a violin built by Stradivari (which would cost up to $4 million) from that of a new American instrument (a couple of thousand). . .

Likewise, Ashenfelter and Quandt (1999) [here in DOC] show that there is lack of concordance between wine judges. Hodgson’s (2008) [here in PDF] result is even stronger, since he finds that only about 10% of the judges are able to replicate their score within a single wine medal group.

In artistic skating, evaluation depends on the incentives and the monitoring faced by judges. Lee (2004) points out that they face an “outlier aversion bias” because they may be excluded from further competitions if they cannot explain why their rating is at odds with the mean of other judges. Therefore, they manipulate their ratings to achieve “a targeted level of agreement with the other judges,” which essentially implies that their judgement is based on previous achievements, and not on the one that is unfolding, since they have to cast their votes a couple of seconds after the performance of each skater.
In related news the Federal Reserve has released transcripts of their deliberations in 2006 on the eve of the financial crisis. NPR provides a nice round-up of coverage:
Here's how the Los Angeles Times frames the story: The transcripts, released Thursday after the usual five-year wait, "reveal in painfully embarrassing detail the high degree of overconfidence and lack of foresight just ahead of the real estate collapse and financial crisis that engulfed the nation" . . .

Perhaps, The Wall Street Journal's Real Time Economics blog found the best bit. During the March 27-28 meeting the Fed's chief economist, David Stockton described a dire situation, which Bernanke acknowledged but quickly dismissed:
"'Right now, it feels a bit like riding a roller coaster with one's eyes shut,' when discussing his forecast for a modest slowdown in housing. 'We sense that we're going over the top, but we just don't know what lies below.' Later, he notes that housing is 'the most salient risk' to the economy. 'I just don't know how to forecast those prices,' he says of housing prices.

"'Again, I think we are unlikely to see growth being derailed by the housing market, but I do want us to be prepared for some quarter-to-quarter fluctuations,' Bernanke says. He identifies housing as a crucial issue, but adds that he agrees 'with most of the commentary that the strong fundamentals support a relatively soft landing in housing."
We are not as smart as we'd like to think we are. Trust me on this, I'm an expert.

16 comments:

  1. I've seen 'outlier aversion bias' in operation on many scientific review panels. It's one reason I tend to discount arguments that cite 'scientific consensus'; scientists, as much as anyone else, are vulnerable to such bias.

    One needs to give expert opinion due credit. As your post emphasizes, due credit should not be over-deference.

    ReplyDelete
  2. As bad as the hubris of experts clearly is, it is not the worst problem. The worst problem is the deference to experts from the media and the rest of society.

    ReplyDelete
  3. On wine experts


    http://scienceblogs.com/cortex/2007/11/the_subjectivity_of_wine.php

    http://www.sciencedaily.com/releases/2009/09/090912124050.htm

    ReplyDelete
  4. I've always felt that humility is an under-rated virtue. Only people who make predictions that are difficult or impossible to verify will have confidence that they really understand nature.

    ReplyDelete
  5. It only took 400+ years for someone to figure out to build DaVinci's helicopter.

    ReplyDelete
  6. Hi Roger,

    This is one reason I find it hard to understand when people support the federal government making decisions on virtually every aspect of peoples' lives. (Such as how many lumens per watt their lightbulbs should be. ;-))

    Which reminds me...;-)

    Have you ever heard of Haitz's Law? You should check it out, if you haven't. Basically, Haitz's Law strongly implies that LED bulbs will be *the* lightbulbs in another 10-20 years. So it hardly makes sense to have light bulb manufacturers futzing around with improvements to the incandescent bulb.

    But I disgress... ;-)

    ReplyDelete
  7. Musicians and wine drinkers - relying on prestige. Skating judges - corrupt. Economists - nuff said.

    ReplyDelete
  8. Expensive wine and violins, among other things, like art and Aspirin, are status symbols. Other people knowing that you can afford something highly valued tends to enhance your social standing. We know there is no justification to assume that the most expensive painting is the best painting. We also know that generic Aspirin works as well as branded Aspirin with a price difference of several 1000%.

    But how does the story about Bernake relate to this? Seems like a different problem to me.

    ReplyDelete
  9. Gerard,

    In the book, The Wisdom of Crowds, one of the important qualifiers is that the inputs be independent. We are all social animals and we all look around us for validation. Committees produce garbage and committees of experts can be the worst.

    ReplyDelete
  10. The role of consensus in science is to justify an action when a system is incompletely characterized and a preferred position is supported through limited, circumstantial evidence. It is in the best case employed as an unscientific means when a risk assessment is incomplete, but otherwise suggests that some action is necessary.

    Official recognition that the real estate market had been distorted occurred as early as 2000. The principal conduits for distortion, the GSEs Fannie and Freddie, were identified, but there was only a minimal effort to hold them accountable, and the legislation never left the house committee.

    In scientific enterprises, as in politics, we need to identify and mitigate the risks that arise from conflicts of interest. This is not strictly a criticism of expertise, but of human nature to employ leverage to improve their position. The criticism should begin with authoritarian interests, followed by expert witnesses, then opportunists and conspirators, and, last but not least, people in the general population that support it in hopes of reaping a profitable return.

    The concerns about experts is only incidental to the positive and negative outcomes their testimony can engender. They have to be held to a higher standard. And while journalists are not experts, but reporters, they disseminate information to their market (i.e. people) and they must also be held to a higher standard. Finally, there are the authoritarian interests, including elected representatives and the perpetual bureaucracy, which has a granted authority to interfere with our lives. They must not be compensated (e.g. COLA) for the failure of their policies. It must be in their best interest to serve the interests of the people they represent, and not their political and material ambitions.

    ReplyDelete
  11. -8-Reiner Grundmann

    Thanks ... how would you suggest adjudicating between the different views expressed between Bernanke and Stockton in 2006? (At the time)

    Seems a lot like how we adjudicate fine wine and violin quality, no?

    ReplyDelete
  12. n.n. Presented very good points. I would like to see evaluations on:

    a) marks given to a same group of students who passed identical exams, but were assessed by different professors;

    b) reviews on an identical research proposal submitted to distinct funding agencies.

    ReplyDelete
  13. Roger 11
    Unlike violines and wine, this is not so much about social status but rather about political communication and management of expectations. Let's assume both had equal knowledge about the risks of the housing market bubble. The question then is what do you do in your respective roles?

    One strategy is alarmism. We know this from other areas such as public health where it can have a beneficial function (climate change seems to tell a different story).

    In the UK, the then chancellor Alistair Darling alarmed the public about a recession (Guardian interview of 30 August 2008): "The economic times we are facing... are arguably the worst they've been in 60 years. And I think it's going to be more profound and long-lasting than people thought." It is not obvious that this was the best strategy. Darling was not able to convert this into political capital despite him being right.

    Think about the real possibility of cash flows stopping in a few weeks time and the social mayhem this would cause. Would alarmism help prevent it? Or would it create a self-fulfilling prophecy? Hindsight is of course the best friend...

    It comes down to politicians seeking to avoid panicky behaviour so they stay in control. They promote the 'soft landing' and try to reassure the public, and above all, the markets. But they should have a Plan B based on what they know about the worst risks. All too often they seem to believe their own hype.

    ReplyDelete
  14. It's been mentioned here before, but David Freedman's "Wrong" is a great source in this area. Here's a quote:

    "Ioannidis did find one group of studies that more often than not remained unrefuted: randomized controlled studies that appeared in top journals and that were cited ... one thousand times or more. Such studies are extremely rare and represent the absolute tip of the pyramid of medical research. Yet one fourth of even these studies were later refuted ..."

    If you infer that the majority of all published scientific papers are likely to be wrong ... you would be right.

    ReplyDelete
  15. Reiner,

    Roger just listed a variety of examples where people who are designated "experts" and held to be such by the public often have no expertise at all. The reasons why may differ greatly, but the bottom line is the same -- view experts with a large grain of salt.

    Especially when the expert pontificates on the future. see Tetlock. See also Future Babble by Dan Gardner.

    The specialization of labor certainly increased productivity, but it also tends to induce within people a tendency to put too much trust in people designated as experts.

    ReplyDelete
  16. If you were running a govt protected ponzi scheme how would you behave? What would your number one job be?

    ReplyDelete