Dear Alastair, 

As you will know, the Office for Statistics Regulation has undertaken two reviews in the last year related to the Scottish Government’s Scottish Household Survey. Each review has had different and distinct areas of focus. This review was prompted when Roger Halliday wrote to us in May 2021 requesting that we undertake a compliance check on the arrangements in place for collecting and reporting gender identity in the Scottish Household Survey. Our other review focused more generally on the quality of the statistics given the decision to switch from face-to-face interviewing to carrying out telephone interviews. 

Collecting and reporting data on an individual’s sex and gender identity or trans history is an area which has drawn much interest and attention from statistics producers, the survey participants and data users alike. We are aware of some of the challenges faced by statistical producers. Often these arise from the sensitive and personal nature of the data in question, particularly when statistics producers are looking to make changes to the data that they collect. 

In recent years there has been several important developments on this topic in Scotland. These include the publication of the Scottish Government’s Sex, gender identity, trans status – data collection and publication guidance in September 2021, the sex question – and related guidance – used for Census 2022 in Scotland, and the changes to the survey questions for the 2022 Scottish Household Survey. We kept this review open until the 2022 Scottish Household Survey questions were finalised to allow us to take these changes into account.

For this review we have looked at the introduction of the Scottish Household Survey’s gender identity question in 2018, including the approach taken by Scottish Government following a 2017 consultation on what changes users would propose to survey questions. We have considered this in the context of the 2022 Scottish Household Survey which has replaced the gender identity question with a sex question and optional trans status question (as per the Scottish Guidance noted above). We have also considered how Scotland’s three population surveys are managed and harmonised in their approach to collecting data on sex and gender. 

In the last couple of years we have seen a much greater understanding of best practice when collecting and reporting data on an individual’s sex and gender identity or trans history by Scottish Government, and more widely across all statistics producers in the UK. However, there are lessons to be learned by Scottish Government on the approach taken following the question consultation in 2017, particularly on matters such as consistency of approach, suitability of interview guidance to support respondents and the requirements to respond on behalf of others. We consider that if the review was carried out today, the outcome would probably not see the same changes implemented. A summary of our findings, including recommendations, is set out below. 

• It is important that users of statistics from the Scottish Household Survey understand the changes that have taken place to the survey and whether these changes have any implications for data quality or comparability with previous Scottish Household Survey statistics. Scottish Government should support users in the appropriate use of these data, particularly where their data needs include making long term comparisons. This information should be provided to users in a clear and prominent manner. 
• Though consultation was undertaken prior to developing a gender identity question in 2018, Scottish Government did not seek feedback on the proposed change before it was implemented. Such feedback should always be proportionate to any change, and it is important that Scottish Government reflects on the importance of this for any future question changes.  
• The Scottish Household Survey is one of three main population surveys in Scotland alongside the Scottish Crime and Justice Survey and the Scottish Health Survey. Since January 2012, a set of 20 core harmonised questions from the Scottish Surveys Core Questions has been asked in these surveys and responses are pooled to provide the Scottish Surveys Core Questions dataset.
• The change to the Scottish Household Survey question in 2018 meant there was not a consistent approach on a core variable that contributes to the Scottish Core Question data in the period 2018-2021. Scottish Government should provide information on this to users of the Scottish Survey Core Question data and explain any limitations and implications for data quality for this period.
• The Scottish Survey Core Question gender question has recently been changed to reflect the Scottish Government’s September 2021 guidance referenced above. This change should be communicated to users as a priority.
• We encourage Scottish Government to consider what further improvements it could make to the broader oversight, management and consistency of approach when it comes to Scotland’s main surveys and the use of Scottish Surveys Core Questions. This is particularly important as the Scottish Government considers how it will support the broader implementation of its new guidance on data about sex, gender identity and trans status. 

Full details of our findings of this review can be found in Annex A. 

Thank you to your team for their positive engagement in carrying out this review and we look forward to continuing to engage with you and your team. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I have copied this letter to Alison Stout, Principal Research Officer, Scottish Household Survey Team.

Yours sincerely,

Ed Humpherson
Director General for Regulation 

---

Annex A Findings from Gender identity and Scottish Household Survey Review

Background and strengths of the Scottish Household Survey

The Scottish Household Survey is an important and valuable data collection exercise. Run by the Scottish Government since 1999, the survey collects data on Scottish households to provide statistics on a wide number of topics including finance, environment, housing, and volunteering. The survey involves a household composition section – completed by the highest income householder or their partner on the composition and characteristics of the household, and a random adult section – completed by a randomly selected adult member of the household on their attitudes and experiences such as satisfaction with local services. 

There has clearly been a commitment and investment from Scottish Government, and directly from the Scottish Household Survey team into the data and statistics gathered through the Scottish Household Survey. Prior to the COVID-19 pandemic, annual user days were held to share developments and plans for the Scottish Household Survey and to better understand user needs. The most recent user day was in March 2019. The range of outputs from the survey are varied and have been developed and improved over recent years. For example, the main annual report was redesigned in 2019 to provide a more digestible summary of key finding and, at the same time, a data explorer tool was launched for users to create interactive tables, charts and reports on survey topic areas. These core outputs for 2019 were accompanied by engaging additional resources including data comics, a podcast and a Scottish Household Survey finding quiz. 

The Scottish Household Survey outputs are accompanied by information for users on methods and quality including on comparability with other sources –  as data on topics such as income, employment or housing, the Scottish Household Survey may not be the recommended source of statistics. As well as the more detailed documents on quality, a more accessible methodology report is available through the data explorer tool which provides non expert users with basic but understandable information on aspects of methods such as sampling and uncertainty. 

There has been a number of changes to the survey question that collects demographic information about respondents and members of their household. Changes were made to survey questions in 2018 and in 2022.  

Summary of changes to SHS questions

2018-2021 surveys 

For the 2018 Scottish Household Survey, Scottish Government implemented a change to the survey question that collects data on whether householder members are male or female.  

This change was implemented following a broader review of user needs across the Scottish Household Survey, the Scottish House Condition Survey and the Scottish Survey Core Questions undertaken in 2017. The review, which aimed to gather views on what changes users would propose to survey questions, was launched via the Scottish Household Survey website and was promoted more widely, for example through existing stakeholder contacts and on ScotStat and Twitter networks. Scottish Government published the outcomes of its review and published its justifications for changes to the survey on its website.  

Following this review, the question was changed from:

2017 Survey

2018 Survey

This question was asked in the first section of the survey where the respondent gives a response for all members of the household.  

The new gender identity question was asked in the 2018, 2019, 2020 and 2021 survey.  

2022 survey 

The Scottish Household Survey team has taken action to implement Scottish Government’s new Data collection and publication guidance – Sex, Gender Identity, Trans Status for the 2022 survey. The survey is currently in the field.  

In line with Scottish Government’s guidance, the 2022 Scottish Household Survey asks the following questions: 

In the household respondent part of the survey

Only in the Random Adult survey questions where one of the adults in the household is randomly selected to complete

While the Scottish Government undertook and published the outcomes of its 2017 review of survey questions, we found that it did not widely consult on the outcome of the consultation or on the resultant changes that were implemented.

Unpicking the Scottish Government’s decision making from 2017 – when there have been many changes to staffing, the policy, and public thinking on matters in relation to sex and gender identity – has been complicated.

When speaking with the team, we consider that then and now, individuals are committed to doing the best for both respondents and data users in its developments around the survey as a whole and on the issue of the collection of data on sex and gender identity. For example, the survey team regularly monitor any issues raised by respondents who may find completing the survey questions in any way problematic.

We consider the outcome of the review in 2017, if carried out today however, would likely not see the same changes to the survey being implemented. This would include decisions on matters such as clarity on the terminology used in data collection and reporting, the questions and response options used in the survey, interview guidance to support respondents, and the nature of responding on behalf of others.

Harmonisation and Scottish Surveys Core Questions

The Scottish Household Survey is one of three main population surveys undertaken in Scotland alongside the Scottish Crime and Justice Survey and Scottish Health Survey. 

Since January 2012, a set of 20 core harmonised questions from the Scottish Surveys Core Questions have been asked in the Scottish Household Survey, the Scottish Crime and Justice Survey and the Scottish Health Survey. Responses on these questions from all three surveys are pooled to provide the Scottish Surveys Core Questions dataset. This increased sample size provides better estimates for data users. 

The current core questions, which were last published in February 2017, include a gender core question.  

The demographics survey questions used in the Scottish Household Survey, the Scottish Crime and Justice Survey, the Scottish Health Survey and the Scottish Surveys Core Questions during the period 2018-2021 are detailed below. 

	Scottish Household Survey	Scottish Crime and Justice Survey	Scottish Health Survey (var. A)	Scottish Health Survey (var. B)	Scottish Surveys Core Question
Guidance/preamble	Ask HA6 for each member of the household Note to interview: Do not probe.	I'd now like to take the name / initials, age and gender of each person in the household so we can see how crime and attitudes vary by different types of household.		If you would like to, please write in the other words you would use below:	Question Title “Gender”
Survey question	How would you describe your gender identity/how would you describe the gender identity of {NAME}?	Are male or female?	[Sex] ASK: Is (name of respondent) male or female?	How would you describe your gender identity?	Are you male or female?
Response options	(1) Man/Boy (2) Woman/Girl (3) In another way (if you would like to, please tell me what other words you use)	1 Male 2 Female	1 Male 2 Female	Man / Boy Woman / Girl In another way	1 Male 2 Female

The Scottish Health Survey introduced an additional question on gender identity in 2018 and retained the collection of data on whether respondents are male or female.  

As outlined earlier, the male/female question in the Scottish Household Survey was replaced in 2018 by a gender identity question. This was treated as the equivalent gender question for the purposes of Scottish Household Survey reporting and onward use in Scottish Surveys Core Questions dataset. Information about the differences in survey approach, on the impacts on data quality and support for interpreting the data, are not currently available to users of the Scottish Surveys Core Questions data.  

While there may be reasons that different surveys might not use harmonised questions, and the Core Questions document states “although in a small number of specified instances the questions may differ somewhat but still provide harmonised outputs”, it is unusual that there was not consistency in approach maintained across the three surveys on a core variable in the period 2018-2021.  

The three survey teams have previously worked closely together with input from the Office of the Chief Statistician. We have seen in more recent times that continued collaboration when it comes to, for example, changes to survey methods given the impacts of the COVID-19 pandemic.  

Nevertheless, we consider Scottish Government could learn lessons from how the three surveys diverged on the questions used to collect data on a core characteristic. Scottish Government has told us it is already taking this forward – for example through reviewing the roles of the different groups involved in the oversight of the surveys and providing clearer communication to users of the Scottish Surveys Core Questions pooled data resource of when it may be the preferred source for certain data types.  

The Scottish Surveys Core Question gender question has recently been changed in response to the publication of the September 2021 Data collection and publication guidance – Sex, Gender Identity, Trans Status. 

In summary,  

• It is important that users of statistics from the Scottish Household Survey understand the changes that have taken place to the survey and whether these changes have any implications for data quality or comparability with previous Scottish Household Survey statistics. Scottish Government should support users in the appropriate use of these data, particularly where their data needs include making long term comparisons. This information should be provided to users in a clear and prominent manner.  
• Though consultation was undertaken prior to developing a gender identity question in 2018, Scottish Government did not seek feedback on the proposed change before it was implemented. Such feedback should always be proportionate to any change, and it is important that Scottish Government reflects on the importance of this for any future question changes.  

• The change to the Scottish Household Survey question in 2018 meant there was not a consistent approach on a core variable that contributes to the Scottish Core Question data in the period 2018-2021. Scottish Government should provide information on this to users of the Scottish Survey Core Question data and explain any limitations and implications for data quality for this period. 
• Recent changes to the Scottish Survey Core Questions gender question should be communicated to users as a priority.  
• We encourage Scottish Government to consider what further improvements it could make to the broader oversight, management and consistency of approach when it comes to Scotland’s main surveys and the use of Scottish Surveys Core Questions. This is particularly important as the Scottish Government considers how it will support the broader implementation of its new guidance on data about sex, gender identity and trans status.