Statistical errors aren’t unusual – so it’s important to measure their effects.

Listener: 13th March, 2014

Keywords: Distributional Economics; Statistics;

There was a bit of flapdoodle recently when the Treasury and Statistics New Zealand owned up to having made an error in some household income statistics, which had a knock-on effect on the Ministry of Social Development’s estimates of the household income distribution. Journalists superficially explained the mistake; their chosen commentators ponderously said it was “significant”; politicians squabbled their predictable political points.

You might not have realised that the ministry alone released 27 densely packed pages of corrections and commentary; or if you did, you may have marvelled at how journalists and commentators had mastered them so quickly. They hadn’t of course, merely grabbing one statistic and focusing on it.

You’ve probably forgotten it all by now, but there are lessons so it’s worth the recall. First, why did the mistake happen? There is a technical answer involving double counting, but there is also a general one. The error was not large enough to be noticed. When professional statisticians are working with data, they are constantly looking for such mistakes. A large one and we rework the data – or are on the phone to sort it out.

Moser’s law is uppermost in our minds: “If a statistic looks interesting, it’s probably wrong.” If it proves to be right, then – hallelujah! – we have a story, but usually it is only a correction. Unfortunately, this error was too small – hardly significant – to be picked up in this way.

Apparently it was found in the process of routine checking. It’s worth making the point that this was happening and the officials published their correction. To err is human, but owning up has an element of divinity. You can trust that if it happens again – it will, but not too often, one hopes – you can expect the same integrity.

Since it was not a “significant” error, I was not surprised when the Treasury said the change in the data had not changed their policy advice. On the other hand, there were journalists and commentators who said the revision meant there were 20,000 more poor in New Zealand. Nonsense. This was a revision to the statistical estimate of reality, not the reality itself. The poor remain poor however we measure them.

For my part, the correction hardly altered my understanding, because the change was within the margins of error. When I – like any professional statistician – am working with data, I am constantly aware that the estimates are not precise and I take that imprecision into consideration. It is why my writing is sometimes a little fuzzy compared with the inexpert who treat any statistic as exact. Sure, a figure may have increased by 0.1%, but there may be no change or it may even have gone down. How often do we see a headline shouting that a small change is important, when the number is later revised as better data come in? Professionals are likely to say “not much change”.

So while the revision does not change my views, it confirms that New Zealand is in the top half of the OECD for inequality, whereas three decades ago we were in the bottom half. It also puts our degree of inequality closer to – but still a fraction less – than Australia’s (although they may be more unequal because of their greater population).

I was disappointed the revision did not add a lot to my understanding of the way the global financial crisis affected income inequality. It is not obvious how it should impact; many people automatically assume that it will be “worse”. But a careful analysis has both rising unemployment adding to inequality and falling profits reducing it. Analysis is complicated by having only a few observations and contamination from the Canterbury earthquakes. My guess is that the effect of the crisis on the income distribution has not been great.

But the basic statistical conclusions remain: income inequality is markedly higher today than it was three decades ago. There is a serious poverty problem that, because it involves children, compromises New Zealand’s long-term development. What to do – if anything – is a political issue.