Affidiavits are usually too specific to be of interest outside a particular court case. However, this one may be of wider interest, because it sets down some issues about reporting surveys to a court. The specific references to the person whose affidavit is being rebutted are removed, because it is the general principles which of interest here. Also removed are the cross references to the earlier affidavit. Otherwise there are no changes. The full affidavit is filed in the High Court, and its identifying details can be obtained from me. Brian Easton.
Keywords: Maori; Statistics;
1. Introduction
1.1 My name is Brian Henry Easton. I am an economist and social statistician. I have degrees in mathematics from the University of Canterbury, and economics from the Victoria University of Wellington, and a D.Sc from the University of Canterbury. I am a Fellow of the Royal Statistical Society and a Charted Statistician. As well as having been a consultant statistician, having written research papers in statistics, and being on two advisory committees of Statistics New Zealand, I have taught statistical methods at the University of Sussex and the University of Canterbury.
1.2 I have been asked to assist the court by reviewing the affidavit of Mr X of the Y company. …
1.3 My basic conclusion is that the Court should be very cautious in giving any great weight to the findings which Mr X reports in his affidavit.
2. The Survey Procedure
2.1 Mr X reports on a survey of 352 Maori over the age of 18 in regard to their attitudes towards a Maori Registration Service.
2.2 The sample of the survey is drawn from those on the Maori and General Electoral Rolls, who have classified themselves as ‘Maori’.
2.3 I am not aware how Maori on the General Roll were identified. Almost certainly some will have been missed including those who did not – for one reason or another – register they were Maori. Additionally there are Maori who are not on the electoral roll – especially young Maori. It is likely that those omitted from the survey would respond differently to the questions. If they did the reported results may be biased – that is not properly reflect the population characteristics (e.g. the proportions).
2.4 The sample was actually of ‘households with at least on adult person who identified as Maori’. Households were contacted and ‘a qualifying respondent selected at random from all those in the household who qualified’. On the face of it, this brings a potential bias into the sample, since a Maori in a household with a single Maori has a higher chance of being sampled than a Maori who is in a household with other Maori. There is a straightforward statistical procedure for correcting this bias (in essence by up-weighting the response of those in multiple Maori households) but there is no mention of such an adjustment. It is more difficult to adjust for the tendency to survey those at home, so those more likely to be out when the call was made – notably the young and the publicly active – are likely to be under-represented in the survey outcome.
2.5 Typically some households choose not to answer or cannot be contacted. (Reasons that they are not contacted are the household has moved on from the last-available address, it is out, or it has no landline.) There is no mention of a non-response rate in the affidavit. Typically non-respondents are different from respondents. This introduces another source of potential bias.
2.6 It is usual to verify the soundness of the sampling scheme by comparing sample characteristics against known population characteristics (such as age and gender). It is not unusual in sampling procedures, such as that used by the Y Company, for the sample characteristics to differ markedly from the population. (I should not be surprised if the sample had a higher proportion of older Maori.) A substantial difference alerts users that the sample may not be a random selection from the intended population, in which case the underlying statistical theory being used to draw inferences may not be applicable. It is sometimes possible to reduce this effect by re-weighting the sample. There is no report of such testing of verification or re-weighting in the affidavit.
2.7 The sampling procedures that Y Company use are often satisfactory for commercial and public policy purposes. However in my opinion, a Court would want a higher standard. At the very least it would want more detail of the sample design and implementation, and some guidance as the omissions raised above and their possible impact on the conclusions.
3. Margins of Error
3.1 Mr X states that ‘results … are subject to a maximum margin of error of plus or minus 5.2% at the 95% confidence level’ (paragraph 10). I am afraid that the statement could be misleading. Unfortunately it is difficult to unscramble the statement in a simple way.
3.2 To clear away two minor points. First, the expression 5.2% refers to 5.2 percentage points. Second, the word ‘maximum’ is misleading, It refers to the situation were the true population proportion 50 percent (and – a matter I return to in paragraph 3.6 – the sampling procedure is rigorous).
3.3 A more precise meaning of the paragraph 10 statement of Mr X is as follows. Suppose the true population proportion was .5 (half or 50 percent), and the sampling procedure conformed to the relevant statistical theory, the in 95 out of 100 (random) samples (of 352 observations) the sample proportion would lie between .5 plus or minus .052, or .448 to .552. This range is known historically as the ‘95 percent confidence interval’ (although statistical teachers are careful to avoid the expression ‘confidence’ when interpreting the interval). If the population proportion had been other than .5 (say it was .6) the width of the confidence interval would have been smaller. Hence the expression ‘maximum’ in the paragraph 10 statement.
3.4 The way confidence interval might be used in the following way. Suppose the sample proportion proved to be .6. Then it seems very unlikely that the underlying population proportion was .5 given that 95 times in a 100 the sample proportion would be in the range from .448 to .552. However it is possible that the proportion could be, say, .6 and through an incredibly bad stroke of luck a properly administered random sample gave such an outlier.
3.5 In making this point, I accept that as a professional statistician Mr X would adopt broadly the same interpretation. I have explained it because a Court is entitled to a more detailed account than the statisticians’ summary expressions. (I would however take issue with the statement which says the ‘true result would be between 44.8% and 55.2%’. For one thing, as explained in my paragraph 3.4, the random sample may be an outlier and the ‘true’ (i.e. population) proportion may not be in that range.)
3.6 An important reason for presenting the ideas more rigorously is they emphasize that the estimates depend upon an underlying theory. If the assumptions of the theory do not apply then the estimates of the confidence interval may not apply. As explained in my Section 2, the sampling procedure is sufficiently deficient to doubt that the assumptions apply closely enough to use the theory in the apparently simple way it was applied. I am not able to provide the Court with just how inaccurate the confidence intervals may be, although I understand that in the much more favourable commercial surveys used for political opinion polls the rule-of-thumb is to double the theoretical confidence interval.
4. The Questioning
4.1 Courts, perhaps more than any other institutions, are aware of how different questions and lines of questioning may lead to different responses. Unfortunately the affidavit of Mr X does not include the exact instructions to the telephone interviewers. Therefore neither I nor, in my opinion, the Court is in any position to judge the responses of the interviewees in terms of the questions posed.
4.2 As best I can ascertain from the reported responses, may well be sensitive to the information given and the questions asked,
4.2 It may also be relevant to mention here that not only may responses vary by the factors mentioned in my paragraph 4.2, but individuals may change their responses following a vigorous public debate.
4.3 To repeat an earlier conclusion. While the survey may have been of great assistance to the commissioners in their development of a policy for a Maori Registration Service, as presented in the affidavit the survey is of much less value to a court.
5. Conclusion
5.1 In my opinion, the presentation of the survey in the affidavit of Mr X is not at the scientific or evidential standard that a Court would require.
5.2 I am also of the opinion, based on the evidence in the affidavit, that the results are certainly subject to a higher margin of error than is claimed, and they may well be biased to an important degree. (In this context bias has only the statistical sense that the sample outcomes may typically some difference from the underlying population parameter which is being measured.)
5.3 On the basis of the material in the affidavit, I conclude that the survey demonstrates some tentative support for a system of Maori registration, although the support may be influenced by the way in which any register is administered, and may change following public debate.
5.4 In summary, my conclusion is that the Court should be very cautious in giving any great weight to the findings that Mr X reports in his affidavit.
SWORN at London by the said BRIAN EASTON in October 2003.