Expert political judgment and the dream team

� Mr Rudd by Colin Wicking

The media is inevitably full of predictions about the Rudd/Gillard Labour leadership. What follows is the case for flipping straight to the sports pages.

Because none of the punditocracy have much of a record of accurate judgment in the week after the election of a new leader. If the punditocracy were to be judged on their assessments of new leaders, most would have a closer acquaintance with Centrelink.

Michael Costello, a bruised Beazley loyalist, angrily makes the point in The Australian (New faces not enough).

John Hewson, Alexander Downer, Mark Latham, Rudd. The media reaction in each of these cases has been the same: “a fresh face”, “the dream team”, “a new generation”, positive coverage, polls shoot up. In the case of Hewson and Latham, they not only lost the only election they stood for but their parties went backwards. Downer never made it to an election, despite rave reviews for many months.

Gerard Henderson has been making the same point for years in his Fairfax columns (Predictions that missed their Mark) and at the Sydney Institute’s Media Watch.

But it was Berkeley psychologist Professor Philip Tetlock who really nails the global punditocracy’s record in 2005’s Expert Political Judgment: How Good Is It? How Can We Know? (Amazon link), a book based on 20 years’ assembling of evidence. Louis Menand nicely summarised Tetlock’s book in a New Yorker piece Everybody’s An Expert:

… People who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake …

… No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.

Or as Tetlock himself puts it:

… The evidence falls into two categories. First, as the skeptics warned, when hordes of pundits are jostling for the limelight, many are tempted to claim that they know more than they do. Boom and doom pundits are the most reliable over-claimers …

… Second, again as the skeptics warned, over-claimers rarely pay penalties for being wrong. Indeed, the media shower lavish attention on over-claimers while neglecting their humbler colleagues.

Tetlock got his 15 minutes, and then nothing changed. Conventional journalistic wisdom stayed concreted in place: bold predictions and fake mind-reading efforts make good copy, or at least the copy that editors want. So, to take an instance at random, Shaun Carney and Michael Gordon in The Age (Back in the game) report Rudd and Gillard are seeking:

“The equivalent of a rebirth of the Labor Party … a comprehensive repositioning of the ALP at the centre of the political debate … the consensus among shrewd thinkers on either side of Parliament is that there is now a serious contest to be had.”

This sort of media content could benefit from a couple of innovations.

The first innovation is a better indication from the journalist about the strength they assign to their own predictions, so that we get writing like this imaginary passage:

“Rudd believes the ALP needs to move closer to the centre of the political landscape (certainty: 95%) … He should perform much better than Beazley at the 2007 poll (certainty: 65% that he will pick up seats).”

The second innovation would be a database of predictions (weighted for certainty, ideally) against which the punditocracy could be judged – a Gruen-style effort to improve the quality of consumer information. Tetlock himself notes that “no society has yet created a widely trusted method for keeping score on the punditocracy.” But it would be instructive to assess the predictions of a Malcolm Farr, a Sean Carney, a Miranda Devine against their previous performance. Suggests Tetlock: “Pundits might adapt to accountability by showing more humility, and political debate might begin to sound less shrill”.

We seem hard-wired to enjoy firm narrative, so don’t expect a media revolution tomorrow; these sort of innovations will take time if they happen at all (certainty: 99.999% that the pundits will not suddenly change their writing styles en masse).

Also possible, though, is that the punditocracy will draw less attention over time, its judgments less important in the face of some innovation that better captures the wisdom of crowds – perhaps a fine-grained system of prediction markets (certainty: no-real-idea-at-all-%).