Saturday, September 24, 2005



What's wrong with the polls

The response rate. People like to laugh at social scientists, but they tend to get very iffy about a survey when less than 70% respond, and they'd regard something with a response rate of lower than 50% as pure garbage unless there were very good reasons to believe that the non-responses were not affecting the sample. And yet, we had polls with response rates between 16% and 65% (and averaging around a third) published and broadcast as if they were gospel.

Not that this is entirely the media's fault - polling companies tended not to report the response rate (the TNS polls of the Maori electorates being a notable exception). And the reason for this is pretty obvious - because if would make it perfectly clear that they were peddling snake-oil. And OTOH, given the Herald's headline - "Erratic election polls blamed on sample size" - you really have to wonder whether they'd notice.

8 comments:

I suspect that in more established electoral systems (US, Britain) the pollsters have developed an accurate set of correcting factors that they can apply to counteract poor response rates (and sample bias).

With MMP only a few elections old, people keep changing their voting patterns and this stops the pollsters developing an accurate system.

I don't know if anyone does old-fashioned door to door or street research any more - this is supposed to avoid biases from telephone ownership and answering patterns.

Posted by Rich : 9/25/2005 06:30:00 PM

Rich: it does - but its expensive, which is why people don't do it.

Posted by Idiot/Savant : 9/25/2005 07:08:00 PM

With great respect to both rich and idiot/savant could I suggest it you don't know about a subject, such as market research ,or polling .to use the election vernacular that you would do us all a service by either finding out the facts or keeping quiet.

Posted by Anonymous : 9/25/2005 07:30:00 PM

Anon: while I'm not a pollster, I'm certainly familiar enough with research methods to spot the flaws in their methodology, as well as the flaws in the alternatives. The big problem with door-to-door polling (other than expense) is that it is inherantly local - probably too local even for an electorate, let alone the country as a whole. Phones get around that problem, but as noted above, the poor response rate of some polls means the results are highly questionable to say the least.

As for market research, when even social scientists snigger at something, you really have to worry...

Posted by Idiot/Savant : 9/26/2005 12:06:00 AM

I did 1st year degree stats, which is a whole lot more than I'd suspect most of the journos writing about polls did.

Posted by Rich : 9/26/2005 09:18:00 AM

I saw one market research guy on television arguing that it didn't matter that the response rate was only about a third because they were confident that the remaining two-thirds would have the same distribution of opinion. What bollocks!

Posted by Anonymous : 9/26/2005 07:49:00 PM

I looked up the YouGov results for the last UK election.

YouGov run a reward based Internet polling system. It's interesting that they get such good results - which possibly suggests they have good correction factors in play. They don't reveal exactly how their polling works for obvious reasons - though they do collect pretty good demographics on their pollees.

Posted by Rich : 9/28/2005 04:50:00 PM

> social scientists snigger

Polling is just an attempt to simulate reality - it is snigger worthy depending on how often and how badly it is wrong compared to other practical methods.

I think if correctly adjusted a telephone poll would be effective and even if not adjsuted they provide informaion about public reaction relitive to polls using the same methodology - useful to politicians anyway.

Also I note that a perfect poll would incorectly predict elections because it is also important to the outcome of elections who will bother to vote and also that some people will make diferent decisions in a poll booth than when confronted at their door.

Posted by Genius : 9/28/2005 08:43:00 PM