The Business Council for Sustainable Development is pushing the results of a survey which shows that 77% of respondents consider climate change to be a problem, as well as widespread support for various policy options (including making emitters pay). But there's a problem: the survey isn't worth the bits it is written in.
The survey was conducted by ShapeNZ - a subsidiary of the BCSD - and was conducted online. People signed up on the website (with the offer of a cash prize as a sweetener) and answered a bank of questions. Their answers were then "weighted to make it representative of the New Zealand population in terms of age, gender, voting behaviour and personal income". This helps a little, but fundamentally you still have a self-selecting sample rather than a random one. And this could significantly skew the results.
Such a methodology would be rejected by any social scientist (unless there were compelling reasons which made random sampling impossible - and then it would be heavily caveated). But it is regurgitated uncritically by the Herald, and is considered to be a useful guide for policymaking. Sometimes I really do despair...
YouGov do something like this, in that they conduct polling from a universe of people who've signed up to be available to fill in questionaires - usually for NZD1.50 or a prize draw entry.
ReplyDeleteThey are apparently reasonably accurate - in that their political polling matches others within expectations and they get a lot of corporate work.
They do have a big (125,000) sample group and do random selection within that group - plus they have pretty sophisticated correction rules.
I meant www.yougov.com BTW
ReplyDeleteThe important difference here is in sample selection - YouGov selects their sample from a representative pool (and I'm willing to accept that 125,000 people is representative enough, at least for most purposes). ShapeNZ's sample is self-selecting, in exactly the same way that a Stuff or TVNZ online poll's sample is. And it's this that makes their results worthless, no matter how much correction they apply.
ReplyDeleteI/S is right. Some other on-line panels are worthwhile because recruitment is random and independent of the surveyed topics. There is then no reason to expect sample bias.
ReplyDeleteHaving looked at the front of the linked website, it seems to link recruitment to the topic. This subjects the results to an umeasurable sampling bias. Maybe they got lucky and the results are representative! But no-one will ever know for sure from this survey.