Friday, September 26, 2008



Overinterpreted

Earlier in the year, I took the Herald's Audrey Young to task over her overinterpretation of polling data. Unfortunately, she's doing it again. In a piece on this morning's Herald poll, she reports that Winston Peters' support base lies in provinces:

[Peters] is favoured by four per cent as preferred Prime Minister in the latest Herald-DigiPoll survey, down 1.6 points.

But only 1 per cent of Auckland respondents favoured him compared with 5.6 per cent in the rest of the country.

We know from the main article that the poll sampled 700 respondents (though it is not clear what proportion of them responded "I dunno"). So what Audrey is saying is that of the 28 people (at best) who said they wanted Winston to be Prime Minister, only three or four of them lived in Auckland. Call me old fashioned, but I prefer to base my conclusions on slightly larger sample sizes than that (still, it could have been worse - she could have been tracking the changes in Winston's Auckland support - "changes" which would almost certainly be the result of pure statistical noise with that size sample...)

This sort of overinterpretation of polls is likely to become rife in the leadup to the election. But it doesn't add to the public's knowledge at all. If journalists are that pressed for copy, here's a suggestion: they could try writing about policy. You know, that thing that politics is actually about, and which the politicians are nominally fighting over? But sadly, it seems our political journalists would much rather focus on the game rather than what is at stake.

Update: Yes, I'm aware that the p-value supports Audrey's overall conclusion. But there's a difference between saying "the poll says that Winston has less support in Auckland than elsewhere" and sticking solid numbers on it; the latter gives an entirely false sense of precision given the tiny sample size. OTOH, as has been pointed out, journalists never qualify their statistics, and if they tried, their subeditors would cut it out, so I'm probably being too harsh here.