Dear Alastair,
Following the decision to switch from face-to-face interviewing to carrying out telephone interviews, Roger Halliday approached the Office for Statistics Regulation (OSR) in January 2022 to provide assurance that the 2020 Scottish Household Survey should be published as experimental statistics and to ask if we could carry out a short review of the statistics.
This is one of two reviews we have undertaken in the last year on the Scottish Household Survey. Each review has different, distinct areas of focus. The focus of this review was primarily on the quality of the statistics given the change in mode, but we also considered the trustworthiness and value of the statistics in the Scottish Household Survey more generally. Our other review focused on the arrangements in place for collecting and reporting gender identity in the Scottish Household Survey.
This review focused on the shift in mode of data collection – that is, the change from interviewing people face to face to telephone interviews. Following our discussions with the survey team and our own research, we found a range of positive features in how the Scottish Government implemented this change in mode:
- The team in Scottish Government engaged with a steering group to discuss plans around publishing the Scottish Household Survey data after the shift to carrying out telephone interviews. The team considered the thoughts of this group as well as discussing with other teams, such as the Scottish Health Survey statistics team. The team used these conversations to help make its decision to publish the data as experimental statistics, leaving out comparisons from previous years.
- We found the publication was very clear in the limitations of the statistics and why they were unable to be compared with those from previous years, as well as providing expert users with a very comprehensive methodology and impact of change in mode document. This helps to reduce the risk of inappropriate use of the data following a change in mode.
- It was also promising to see the National Performance Framework, which gets some of its estimates of National Indicators from the Scottish Household Survey, was also heavily caveated and details the conclusions that can be drawn from the data.
- In order to provide more accessible and insightful outputs, we understand that the survey team is considering how it wishes to develop the Scottish Household Survey and is considering splitting the publication into separate reports. This is a positive step, and we encourage the survey team to engage with users to understand their needs as it develops the outputs.
There are also some key issues we identified with the 2020 Scottish Household Survey that the team might consider for future outputs:
- Due to the relatively low number of participants in the 2020 survey , the Scottish Household Survey was unable to provide local authority breakdowns. It is important that in future years, every effort is made to ensure that there are a suitable number of data points to provide users with such breakdowns to support local authorities who use these data for benchmarking and planning. It is unclear whether this low response rate is a result solely of the COVID-19 pandemic; the impact of the shift to telephone surveys; or a mixture of both. We expect the Scottish Government to monitor the potential impact of the change to telephone surveys on response rates.
- There was an apparent large shift in tenure of people who completed the survey, which was likely to be down to non-response bias rather than actual shift in tenure. Again, it is unclear whether this shift is a result of the pandemic or a direct result of the shift in mode to telephone interviews. It is important that the Scottish Government investigates this potential effect further. We are pleased to hear about the Scottish Household Survey team’s plans to collaborate with other survey teams to discuss the impact of tenure on survey results and improve your network of methodological experts to help you understand this further.
The Scottish Household Survey team appears very focussed on continuing to develop the statistics, and the 2020 Scottish Household Survey will help provide experience in carrying out telephone surveys. It is important that the Scottish Household Survey team continue to improve its collaboration with other survey teams, methodological experts and colleagues to help address the issues found within this review.
Thank you to your team for its positive engagement in carrying out these reviews and we look forward to continuing to engage with you and your team. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.
I have copied this letter to Alison Stout, Principal Research Officer, Scottish Household Survey Team.
Yours sincerely,
Ed Humpherson
Director General for Regulation