Monday, October 31, 2005

Some quick and dirty statistics on Question Time

Last week I posted about Parliamentary statistics, suggesting that we needed British-style participation tools (in the style of the Public Whip) to keep track of whether our elected representatives are actually doing their jobs. One idea was to take a look at Question Time, the time each day when the Opposition is supposed to hold Ministers accountable to Parliament, and (by extension) the government accountable to the public. Who is asking these questions (and more importantly, who isn't) may provide useful insight as to which MPs are worth their salary.

However, it's not quite so simple as that. Two factors which weaken the link between "number of questions asked" and job performance are:

  • Ministers answer questions, rather than asking them, so they will be systematically under-represented (and conversely, the "patsies" they get to ask soft questions so they can attack the Opposition will be over-represented); and
  • While questions are distributed to parties on a per-capita basis (meaning that every MP theoretically has the same chance of asking a question), they are not distributed equally within parties. Generally, the party leaders and front-bench get to hog the questions - and these positions reflect ability in political infighting and faction-building as much as any ability to hold the government to account. That said, a long-serving backbencher who never asks the government anything is probably a strong hint that they are a talentless time-server.

So, Question Time is more a guide to identifying systematic underperformers than anything else. But then, those are exactly the people we want to identify.

Last night I built a crude parser to extract the data on primary and supplementary questions. Unfortunately, the Hansard Office uses Word (bletch!) to produce their HTML - making it difficult to parse, and meaning that there are definitely some errors in the resulting data. But looking at the output and the various flags raised, it seems that the error-rate was less than 1%, and mostly corrected with manual checks. However, there were still around 5 questions (out of more than 2600) which could not be attributed. The sample covered Question Time from 12th February 2003 to 2nd of August 2005. The results are in this spreadsheet. I've identified some ministers and Parliamentary officers to provide context for their scores.

Focusing on the opposition parties, there are a few notables that stand out. ACT's Gerry Eckhoff and the Greens' Mike Ward both asked far less than their compatriots. Fortunately they are no longer with us, and neither are most of NZFirst's tail (with the notable exceptions of Pita Parone and Doug Woolerton). Unfortunately, all of National's notable underperformers - Shane Ardern, Sandra Goudie, Pansy Wong, and Lindsay Tisch - are still with us, and all hold positions of responsibility in National's new lineup.

It would be nice if, for the next term, we had a searchable database with both statistics and links to the questions asked, so we can see what our MPs are actually doing. But unfortunately, I just don't have the time...


What does the comment next to GEORGINA TE HEUHEU refer to?

Posted by Anonymous : 10/31/2005 05:09:00 PM

mmm.. i have a bit of a problem with name and shame on this one.. most teams are comprised of members with different skillsets and you'd expect the more extroverted to be lining up to ask questions, whilst the more introverted play other roles.
Wouldn't this just encourage politicians to play to the camera (or the written record) more, rather than finding the role they do best and getting on with it?

Posted by Anonymous : 10/31/2005 05:52:00 PM

Anon: Due to the way I wrote my parser, I suspect it failed to pick up her questions; However, they are unlikely to number more than a handful, as it did notice when a question time was short, and those were manually checked.

Basically, I think there were about 5 questions where it returned a sentance fragment or a blank space rather than a name. At some stage, I'll probably refine it (especially if anyone wants to set up a proper site to track this); at the moment, tracking down the problem will take longer than my available procrastination time.

Posted by Idiot/Savant : 10/31/2005 06:33:00 PM

Huskynut: DPF makes much the same point here. I'd agree that this is by no means a complete picture, and more information is needed. At the same time, if an MP starts showing up at the bottom of all the metrics we use, we're entitled to ask what they're doing to earn their salary.

Posted by Idiot/Savant : 10/31/2005 06:36:00 PM

Actually I downloaded a whole bunch of Hansard transcripts with a similar view to parsing the HTML and ran into the same problem of it being microsoft-based. I was attempting to do a statistical analysis of the voting patterns of the parties to see how the parties actually relate to each other when it comes to voting. A preliminary analysis was looking interesting... maybe I should give it another go. I wanted to build a political web like this one.

Posted by Alexei : 10/31/2005 06:43:00 PM

This is something I've been interested in doing for a while, but with the same problems parsing the HTML when I gave it a try. Given enough time to tweak though it shouldn't be impossible. More problems turn up when the code is just plain wrong, though.

If somebody else wants to give it a solid crack, I'm happy to provide the server space and/or get it hooked into the conscience votes database (which is going to get an overhaul too).

Posted by Anonymous : 10/31/2005 07:23:00 PM

I/S - fair call. At the other end of the spectrum you have to look at the cost of questions asked. Within govt depts, pretty much everything else is dropped if required to answer questions, which means each question has a cost to the taxpayer. A pile of random and ill-conceived questions are as useless as silence (but more costly to the nation). So you really need a means of correlating questions with useful results (enquiries, policy reviews, whatever) to determine who's earning their cash.

Posted by Anonymous : 11/01/2005 07:39:00 AM


Interesting stats. If you're interested, I did a similar exercise looking at the participation of party leaders in QT over the past 25 years. It is published in "Political Science", the journal of the NZ political studies association, December 2004 (Vol 56, No 2). It is also available at:

Rob Salmond

Posted by Anonymous : 11/01/2005 07:55:00 AM