Guest Column - What Does The 2004 Living Standards Report Tell Us?

By Brian Easton.

The New Zealand Living Standards 2004 report depends entirely upon its "Economic Living Standards Index" (ELSI), first used in the previous (2000) report. At that time I expressed reservations about the index. Many have not been addressed. What I do here is set some down again, and then mindful of the ELSI's problems, and try to draw some conclusions of what the latest survey may be telling us about what happened between 2000 and 2004.

This may all seem a bit tedious. Who cares if the measure does not have any real meaning? Is it not better to use it for whatever (political) purpose we want without worrying about what the measure actually means? Don't we do that all the time? Do we need to have a detailed understanding of the meaning of time or income or whatever? Cant we just trust those who constructed the measures?

Well actually no. Typically scientists have spent much effort in constructing valid indexes. They are usually based on sophisticated theory, while each measure is verified by other scientists and is shown to have a practical significance related to things outside the narrow confines of the data which generates the measure. This is so routine for our authoritative measures that we assume it is true whenever someone proposes a new one. It may not be. Often indexes (i.e. measure of something or other) are not validated and amount little more than the opinions of the proponents.

Validation usually involves two stages. First, there is the underlying theory. Is it rigorous? How does the index relate to it? (Its construction often has to make compromises, but if the theory is well understood one can evaluate the extent to which they matter.) Second is a empirical validation. Does the index relate to anything outside the data from which it was constructed? (Two examples of my trying to empirically validate indexes – sometimes called ‘calibration’ – are at http://www.eastonbh.ac.nz/?p=229 and http://www.eastonbh.ac.nz/?p=460.)

Validating the ELSI Index

The ELSI is based on asking households (strictly an "Economic Family Unit" – EFU) a set of questions about what items they have or have not got, what restrictions there are on their social activities, what economising behaviour they have had to practice, and how they rate themselves. The household responses are then combined to give the ELSI index.

Note there are two stages. The first involves the questions asked. Are they the right questions? The second involves the aggregation of the responses. Has the right weighting (significance) been given to each response? How do we know the selection and weightings are not merely the opinions of those who construct the index, and that another set of "experts" would make different decisions? (For instance, the ELSI doubles the significance of some responses relative to others. Why not three times? Why not half?)

As far as I know, there has been little attempt to validate the ELSI. There is no reference to a satisfactory validation in either the report nor its bibliography. That does not mean the index is necessarily invalid, but we need to be most cautious when using it.

In my view the ELSI is a very poor measure of overall economic living standards (whatever that means). In particular, it is unlikely to be much use discriminating between those who are comfortably off. I should not be at all surprised if Bill Gates and myself would get much the same score, as might someone on the average wage and myself. That is because the questions are not designed to discriminate between the affluent. (There is not even a question on car ownership.)

Thus one can give no significance to the average ELSI for the whole public. (If everyone got a(n extra) car it would have to have no effect on their ELSI score – other than perhaps through changes in self-rated satisfaction.)

The difficulty arises because the researchers were focussing on the circumstances of the poor. So while we should dismiss the ELSI for the population as a whole we cannot be so dismissive of the index for those near its economic bottom. It is important if some people do not have a good pair of shoes (supposing that means "suitable for general purposes", rather than "for dressing up"), which was one of the questions.

So what I am going to focus on are those the authors describe as being in some or greater "hardship". That category includes those in "severe hardship" and "significant hardship". In my judgement their samples are too small and the measurement error too big for one to make comparisons involving them

Comparing 2000 and 2004

Famously, Moser's Law says that if a statistic looks interesting, it is probably wrong. Despite the rise in real incomes between 2000 and 2004 (which the report acknowledges) the ELSI decreases slightly. Which is surely "interesting", (I don't have a feel of the significance of the fall in the ELSI from 40.6 to 39.7 – out of 60. That is one of the consequences of not validating the index.) But given that the index is probably meaningless for the majority – say top two-thirds – of the population measures based on the whole population (such as the mean) are of little value.

What is more disturbing is the evidence that more of the population (that at the bottom) were in the hardship category. The rise in the proportion of the population in hardship rose from 23.6 percent in 2000 to 24.0 percent in 2004.

A useful table (C10) in the report, which gives the proportions for the total and a huge range of sub-populations, shows this difference is not statistically significant – that is it could arise from the vagaries of sampling.

There are a few cases among the sub-populations where there are increases and a handful where the difference is statistically significant (but remember that will happen by accident on occasions, given the way that statistical tests operate). You can pour over the table, and find some example that suits your assumptions. (For instance Labour-committed supporters will mention the decrease in hardship of the proportion of the Maori and two parent families. Those on the other side may draw attention to that there seems to be more employed and more of the elderly in hardship.)

At this point we could drop the whole exercise, concluding the changes in the ELSI over the four years tell us nothing. But there is a really interesting problem – one where the "probability" in Moser’s Law warns us sometimes happen. Sometimes a statistic can be interesting and not wrong.

The problem is this. The four years between the two surveys were ones of economic prosperity by almost all conventional welfare measures – higher real incomes and employment, lower unemployment ... – as well as common sense. Yet there is no improvement in the ELSI. The inconsistency is sufficiently strong to suggest that the ELSI is generally not valid. I have already argued that is true, so I am not surprised. At which point we could end the story.

What interests me though, is that where the ELSI might thought to better represent living standards – among those in "hardship" – there is no improvement either (indeed, a statistically insignificant deterioration). I find this much harder to understand. The ELSI cannot be that hopeless.

I tried to think of some technical reasons. Perhaps household fragmentation has had an effect. When a poor couple splits up only is one is likely to have warm bedding (one of the questions asked) at best. Perhaps the drift north has had an effect too (they were asked about having a warm winter coat). But such effects are likely to be trivial in the four year period.

The report acknowledges that beneficiaries had no real increase in their standard benefits over the four years, and some had reductions because some assistance (such as for families) was not adjusted for inflation. Would their lower real incomes be enough to increase their hardship and reduce the ELSI? Possibly.

There is also a tricky problem that the sub-populations change over time. Suppose a better-off beneficiary in 2000 joined the workforce by 2004. That would lower the ELSI of the beneficiaries who are left. It may also lower the ELSI scores of the employed since they ex-beneficiaries are likely to be at the lower end of the standard living of the employed. Bother, bother, bother.

There is another phenomenon which the ELSI does not deal with well. In each survey, respondents were asked whether they had a personal computer (for example). Now the options (to simplify a little) is "yes", "no" (because I cant afford it), or "I don't want one". The "don't want ones" are then netted out. Now it seems possible that some people who said "don't want one" on 2000 said "no because I cannot afford one" in 2004. Their ownership has not changed one iota, but the standard of living of the household measured on the ELSI will have decreased. Are there enough of such items in the survey to actually depress the score? Possibly.

What too about the family that has bought a computer because they now want (or think their children need) one and have made other sacrifices to fund the computer? The way the index works might depress the score. Indeed the additional purchase need not even appear in the survey. A household which bought a car (not in the survey) and sacrificed their annual holiday (in the survey) to do so. That would definitely depress the score. (And to complicate the story further, what if the car was necessary to go to work?)

Trickling Down Prosperity?

At this point one might become so uneasy about the ELSI, to judge of it of no value. However I think there is an interesting conclusion from the previous couple of paragraphs. To put it provocatively, during a time of prosperity, expectations may trickle down faster than income.

This proposition brings together two issues. For more than thirty years I have been advocating that we should use a relative poverty measure. This expectation effect illustrates its relevance.

Second, for some groups – the elderly and children – there is no automatic market trickle down effect, and that sharing prosperity requires some active intervention by the government.

The elderly have a trickle down through the (imperfect) indexation of New Zealand Superannuation to wages. Beneficiaries don't, because their benefit is indexed to consumer prices which rise slower than wages.

Children have suffered even more because their public assistance is not indexed at all. (There is a big lift in the incomes of some of them via the Working for Families package.)

I have argued that the benefit of the rising prosperity to those on social security has been to get an income increase from a job (not all of which disappears in the costs of employment and taxes and benefit abatement). Those on low earnings may also benefit from longer hours, higher pay and better working conditions, and upskilling. As pointed out, the ELSI can miss such improvements.

But I have to modify that conclusion. What the ELSI for low income people seems to be saying is that those that have not got a job have not benefited. Even if their real incomes are maintained, they may experience falling "living standards" if prosperity raises expectations but not their incomes.

The Real Issue

We can go on in this sort of speculation, but the first real lesson is not to trust changes in the ELSI which contradict common sense. However the study affirms one old truth, and challenges a more recent one.

We have long known there are some groups in hardship. According to the survey over 35 percent of our children are (compared to less than 25 percent of the population as a whole, including children). Interestingly, the hardship measure suggest proportions similar to those in the income poverty studies – the vast majority of the poor in income terms or in hardship in this study are children and their parents. (See various papers indexed in http://www.eastonbh.ac.nz/?p=152 for more details.) Perhaps this should be no great surprise, although it is satisfying to a researcher to have a rather different method coming up with similar conclusions.

The degree to which the government's recent measures to increase family assistance will markedly reduce hardship remains uncertain, because the support has been targeted on working families, rather than all families.

Which leads to a challenge to a more recent "truth". I think it was right in the early 1990s to place some emphasis on beneficiaries finding jobs. The conclusion, dimly seen through this report on living standards, is that after 15 years of doing so, plus a period of employment expansion and prosperity, there remains a rump who are still trapped into hardship. The policy strategy, which has succeeded for many, has failed them. If we are concerned about their hardship, and the often serious and longstanding (even inter-generational) deleterious consequences, we need to think about an alternative strategy which may be more income. It almost certainly is not to intensify a strategy which has failed them in the past.