ONS Information Paper, June 2009
Further question testing ahead of the recommendations for 2011 Census
Since the finalisation of the questionnaires for the 2009 Census Rehearsal, a number of issues have arisen as a result of the parliamentary scrutiny, public debate and engagement with stakeholders resulting from the publication of the White Paper ‘Helping to shape tomorrow’. Subsequently ONS has further considered concerns raised during the period since the White Paper was published, and has identified a small number of issues for further investigation and evidence gathering.
As a result of this there are a few questions that are undergoing further development and testing. Once the testing is complete a decision will be taken about whether any of the new questions being tested should be recommended to Parliament for implementation in 2011.
Currently, ONS is testing or is committed to test the quality and understanding of three specific topics: religion, citizenship and collecting student’s term time address using the second residence question. Testing of the addition of a Kashmiri tick box to the ethnic group question is also planned.
Religion
ONS is currently testing a new question on religion that asks ‘Which of these best describes you?’ with no changes to the ‘No religion’ and pre-designated religion tick-boxes. This is in response to concerns that the recommended question ‘what is your religion?’ will lead to people identifying with a religion when they have only a loose affiliation and that such information will not be the best for user needs.
The testing programme has been designed to gather evidence to investigate the impact of this wording and to look to address issues such as
· the most appropriate wording for the ‘other’ category
· the optimal ordering of the tick boxes
· whether a vaguer question elicits response on concepts other than religion
· whether the new wording changes the distribution of responses to the different options
A postal test will look to assess the distribution of responses against previous question testing and the 2001 Census. The question being used in the postal test is:
Testing in the Opinions Survey will use slight variations on the response options in each wave to detect differences in response. In particular as the question is not explicitly asking about religion and could be implicitly asking about religion or belief, it would be appropriate to be clear that the write-in space is for religious and non-religious beliefs. However, introducing the term ‘belief’ can change how people understand the question and a much more general write-in instruction may also need to be tested.
Qualitative testing will help to interpret the results from the other testing by providing direct respondents’ views on the question and their thought processes for choosing a response.
This testing will also help evaluate levels of measurement error to help assess data quality of the question.
Citizenship
ONS is reconsidering whether to use the word ‘passport’ or ‘citizenship’ within the question. The concern is that with the new migration questions, intention to stay and year of arrival, a question on passports could be seen as a potential barrier to response.
An analysis of item response rates, response distributions and write-in responses from the postal test will provide an indication of any quality issues and the extent to which respondents understand the term citizenship. The question used in the postal test is below.
The revised question on citizenship has also been included on the April and May waves of the Opinions Survey. The aim of this testing is to provide further evidence of respondents’ understanding of the term citizenship and therefore its potential for successful inclusion on the questionnaire.
The qualitative testing will help to interpret the results from the other testing by providing direct respondents’ views on the question and their thought processes for choosing a response.
This testing will also help evaluate levels of measurement error to help assess data quality of the question.
Students and second residence
Further consideration is being given to using the second residence question to collect students’ term time address from their parents at their ‘home’ address. This is being considered as a possible source for helping to understand the number and location of students that could be used within the Quality Assurance of the Census population estimates. Students are a group that were difficult to count in 2001 and this may be exacerbated in 2011 as Census day, 27 March 2011, falls outside of term time in some universities.
The postal test will provide some assessment of how well the second residence question is answered by parents at the home address; the impact on student response from areas outside of term-time; and, whether the routing and ordering of the question has any impact on overall response quality.
Again, some qualitative testing has been conducted to assess respondents’ understanding of what the questions are trying to achieve, and to record and assess the quality of the data being collected.
Ethnic group response category testing
ONS is also carrying out testing to evaluate the performance of an additional tick-box in the ethnicity question, to collect information on people identifying as ‘Kashmiri’. A Kashmiri tick-box already scored highly in our prioritisation exercise.
This will include an element of qualitative and quantitative testing, such as focus groups and a postal test.
ONS is not considering any other new testing on the ethnic group question.
ONS
June 2009