2. What we found so far

2.1 User need for data on gender identity was identified as part of the Office for National Statistics’ (ONS) public consultation on proposed topics for the 2021 Census questionnaire in England and Wales, carried out in 2015. In particular, the ability to capture data on the transgender population was seen as a data gap for service provision that was important to address.

2.2 The 2018 White Paper Help Shape our Future: The 2021 Census of Population and Housing in England and Wales highlights the complex issues to consider in designing a question on gender identity, including the need to clarify how the concepts around sex, gender and gender identity are expressed and can be measured.

2.3 In developing the gender identity question for the 2021 Census, ONS looked at international best practice and worked closely with other National Statistics Institutes (NSI) to understand what questions were already being used and how they performed. These NSIs faced similar challenges to ONS, and this continues today, there now being multiple approaches used to collect data on gender identity in censuses across the world.

The question requires more tailored testing

2.4 Prior to the first publication in January 2023, ONS published a comprehensive article on Sex and gender identity question development for Census 2021 which sets out how it researched, developed and tested the gender identity question. The article provides relevant context on the research that led to the 2018 White Paper recommendation to include a question on gender identity, as well as the research underpinning the question design.

2.5 Some users of these statistics told us that the engagement of experts in the question design and testing was not as wide as it could have been. The ONS team involved in the question development told us that, while they endeavoured to balance competing user needs, it was extremely challenging to identify a question that would be acceptable for use and also addressed the findings of the quantitative testing. ONS recognised from the outset that the chosen approach may need to be updated in time, as societal understanding of gender identity evolves and once there is sufficient evidence to evaluate the performance of the question. This rationale aligns to ONS’s experience of developing questions to collect data on ethnicity, which are now standardised and used widely.

2.6 The transgender and non-binary community represent a relatively small proportion of the population, and therefore the likelihood of this community being picked up in a random sample is low. To address this, ONS carried out some targeted testing of the gender identity question within this community. This included cognitive interviews with transgender and non-binary participants, as well as informal interviews at LGBT+ events. There were no transgender participants in the Welsh language focus groups and cognitive testing. However, ONS noted it received feedback from stakeholders working in Wales who supported the translation based on their experiences.

2.7 In our engagement for this review, we heard from a user who felt that those who are transgender and non-binary have been responding to the Census and other surveys historically in the way that best reflects their identity. They also felt that the risk of miscomprehension is more likely to occur with those not in the trans and non-binary population, potentially including other minority groups. While ONS paid particular attention to capturing those who are transgender correctly, there is a greater risk with false positives or negatives when the target group is small. This is because the effect of incorrect responses on the data will be amplified.

2.8 The gender identity question was also tested with respondents whose gender identity corresponds with the sex registered for them at birth through user experience (UX) and panel-based quantitative testing. The 2021 Census questionnaire as a whole was tested through a Census rehearsal in 2019 involving approximately 300,000 households across England and Wales. This included households in areas that have a high proportion of people whose main language is not English. Feedback from the rehearsal showed that some respondents found the gender identity question difficult to answer, but the non-response rate for this question was lower than ONS expected.

2.9 As part of an in-depth review of measuring gender identity carried out by ONS and Statistics Canada in 2019, ONS shared its findings that, in languages where the concept of gender is not well established, it is unlikely that one standard question could be directly translated and advised that question testing should ensure respondents understand what is being asked. The concept of gender is often conflated with the concept of sex and the term ‘gender’ can be used by some interchangeably with the term ‘sex’. The way these terms are used in law also differs across countries. ONS told us that it faced difficulties finding people with lower English language proficiency who were willing to participate in question testing.

2.10 Translated copies of the Census questionnaire were available in nearly 50 languages and a public support team was put in place to support user queries. ONS also carried out extensive community engagement across England and Wales to promote the Census and encourage different communities to complete it. It told us that the number of downloads and queries received were minimal. It is difficult to know the true scale of their use, as individuals may have downloaded a translated copy on behalf of an organisation or group. Moreover, it is possible that some individuals chose to use online translation tools to translate the English or Welsh questionnaires as they went through completing it. ONS told us that it initially had a small project looking at how all questions appeared through Google Translate. However, once it was decided that ONS would not promote the use of Google Translate, this project was stopped.

The question design involved UK wide collaboration

2.13 A UK harmonisation group was set up to ensure developments and learning were shared across the relevant census statistics producers. This was in line with a published statement that was updated throughout the course of the censuses. Limited user need was identified for a question on gender identity in the 2015/16 Northern Ireland Statistics and Research Agency (NISRA) consultation for the 2021 Census. As a result, no gender identity question was included in the 2021 Northern Ireland Census.

2.14 ONS and National Records of Scotland (NRS) worked closely together throughout the development and testing of the gender identity question, and some of the testing was jointly funded. NRS told us that the engagement with ONS has been excellent throughout the development, testing and now quality assurance of the gender identity data. With a view to creating coherent statistics, the harmonisation group agreed that it would look to ask the same question (the question ONS ended up with) across the censuses if this was appropriate for user need in each nation.

2.15 During the testing, NRS identified specific user need for an explicit transgender question, and one that captured a trans history. This user need was not mirrored to the same extent in England and Wales. Additionally, ONS’s testing received some negative feedback from individuals whose gender identity corresponds with the sex registered for them at birth and non-binary respondents on the trans history question.

2.16 NRS’s testing identified a need for the sex and gender identity questions to be positioned together to provide better understanding by respondents. It also found that a non-binary sex question produced less item non-response than a question set comprised of a binary sex question followed by a gender identity question. By contrast, ONS was guided by the practical consideration of where the question would best fit on paper. It decided that the gender identity and sexual orientation questions should be placed with the questions that only apply to people aged 16 years or over, at the end of the sociocultural questions, rather than immediately after the sex question.

2.17 The Equality, Race and Disability Evidence Units in the Welsh Government published their strategy in 2022. One of the priorities identified was the need to review the extent to which the gender identity question used in the 2021 Census (alongside other options including the NRS approach) meets the needs of Welsh users. The Welsh Government is in the early stages of scoping this work and will be considering how to maintain some level of coherence across the UK. A 2021 report outlining recommendations for furthering LGBTQ+ equality in Wales recommended that the Welsh Government should take practical steps to provide recognition of non-binary identities where legal gender is not required, including reviewing data management to ensure that open-ended nonbinary options are included. The harmonised standard has therefore not been considered acceptable by some LGBTQ+ stakeholder groups in Wales.

2.18 The UK harmonisation group, which includes the Northern Ireland Statistics and Research Agency (NISRA), is working closely to determine how it will collectively produce estimates for Great Britain given the divergence in approach. This work will need to consider the coherence of the data in outputs within nations as well as across Great Britain.

The quality assurance was proportionate based on what was known

2.20 More topic-based quality assurance was carried out for the 2021 Census than in previous years. Gender identity and sexual orientation (combined) formed one of 13 topics assigned for bespoke quality assurance reports. The team carrying out the quality assurance worked closely with the relevant topic leads in ONS to apply expert insight to the data to test thinking. These quality assurance reports were reviewed by an independent quality assurance panel comprising of experienced researchers to apply additional scrutiny to the data.

2.21 The Code of Practice for Statistics states that quality assurance should be proportionate to the data. ONS told us it analysed relationships between the gender identity data and other variables based on areas that were perceived to be of public interest at the time.

2.22 The Census Quality Survey (CQS) is a telephone survey carried out following the Census that asks a sample of people the same questions as were asked on the Census. The CQS in 2021 received usable responses from 8,598 households. Agreement rates are calculated by comparing the answers to this survey with those given by the same respondents in the Census. CQS agreement rates can be a helpful tool for understanding the performance of new questions, particularly if they are voluntary. However, the change in survey mode should be considered where it may have an effect on more sensitive questions. ONS only had access to the weighted CQS agreement rates after the first release of gender identity statistics. In hindsight, it could have prioritised analysing the CQS agreement rates before releasing the data given it was a new question. Moreover, a write-in response was not required for the gender identity question in the CQS which could impact the way the gender identity response was then processed and coded.

2.23 In June 2023, ONS published information on the methodology for collecting and processing data on gender identity in Census 2021. A write-in response for gender identity could only be provided where individuals recorded their gender identity as being different to their sex registered at birth (i.e. answering the gender identity question with ‘no’). Where ‘man’ or ‘woman’ were given as the write-in response for those who answered the gender identity question as ‘no’, these responses were coded to ‘trans man’ or ‘trans woman’. On the online questionnaire, if someone selected ‘yes’ to the gender identity question, any write-in response would be deleted.

2.24 In the article, ONS explains that the coding of write-in responses did not make use of information from the sex question. This was because while the gender identity question refers to sex registered at birth, the sex question referred only to ‘what is your sex?’ As some transgender people may view their sex as aligned to their gender identity rather than sex registered at birth, they may have chosen to respond accordingly irrespective of the question guidance. This is particularly likely to be the case for those who have legally changed their sex and hold a Gender Recognition Certificate. The 2021 Census dataset for gender identity by sex registered at birth shows that the majority of people coded as transgender recorded their sex registered as birth the same as their gender identity. For example, 68 per cent of those recorded as trans men recorded their sex registered at birth as male. This correlation featured in the concerns raised with us and has been perceived by some as suggesting wide misunderstanding of the gender identity question, as opposed to an intentional decision by respondents. These concerns suggest that some users may assume the gender identity question does relate to the sex question.

ONS’s research must communicate how the data can and cannot be used

2.26 ONS has published a range of datasets on gender identity data combined with other variables. These datasets represent a mix of tables pre-determined by user need from the Census topic consultation and bespoke tables produced in response to user need. The datasets range from gender identity by other personal characteristics to qualifications and family composition.

2.27 Alongside the release of these datasets, ONS published a blog in April 2023 which advised users to look out for future articles exploring the data, which would help users interpret the data and understand interactions between variables. While this gives some indication that further work was required to understand the data, it does not adequately highlight the inherent uncertainty in data on a new and hard to measure Census concept, in an area where very limited comparable data was available for quality assurance.

2.28 ONS’s approach to releasing data on gender identity was made in the spirit of transparency and it was keen to deliver value to users by providing new data on gender identity. We recognise that ONS considers it has been transparent about the potential quality issues through the range of updates it has published, set out in the timeline in table 1. The quality information published alongside the first gender identity outputs included an acknowledgement that responses may be influenced by different interpretations of the question. ONS has also introduced more caveats alongside the data following our letter on 19 June, where we recommended that the communication of uncertainty should be strengthened.

2.29 When ONS began receiving queries following the release of first outputs in January, it set up triage meetings that allowed the relevant teams across the organisation to join up and share their perspective on the queries, to provide coordinated and coherent responses to users. This also enabled ONS to respond to media enquiries more easily when these concerns began to feed through into the media coverage of the data. In our interactions with ONS as part of this review, we have seen an openness and recognition of these concerns from the teams in ONS.

2.30 In a context where there are a range of concerns around the data, it is important that users are kept updated on research plans and findings as they emerge to support their use of the data. We recognise the positive intention and ambition that underpinned ONS’s approach to releasing data, but we consider that ONS should have developed its quality information more fully, including clearly and prominently highlighting areas of uncertainty in this new collection, before releasing the data to support appropriate use. When releasing big data sets for the first time, it is inevitable that users will want to interrogate and analyse the data for their own purposes. ONS could have done more to see this as an opportunity to draw on the analysis of external users and be clearer that it was open to feedback as part of releasing the data.

2.31 The content and tone of the published communication from ONS have been somewhat closed and at times defensive. For example, the default position appears to be that the question is the best that was tested, which can still be true while acknowledging the question may have faced some issues when used in practice. It would be reasonable for users to expect a new question with little data available for triangulation to be considered and communicated as experimental and under development. ONS has assured itself that the data appear to triangulate with other sources at the national level, which will be sufficient for some users who only require high level data, but it should consider how it supports the onward use of the granular breakdowns. There remains an absence of an overarching message that explains the nature of this uncertainty and that clearly sets out that ONS is still trying to understand the data through further analysis. There appears to be some reluctance from ONS to communicate uncertainty unless it can be quantified.

Table 1: Timeline of ONS updates on gender identity.

2.32 ONS has said it is focusing on developing a comprehensive package of research on gender identity that explains to users what can and can’t be done with the data. This will be informed by conversations it is carrying out with key user groups. It told us that it has chosen to delay publishing its findings until it can be more confident in the conclusions it is drawing. We recognise the complexity and sensitivity of the research ONS is conducting and agree that this should not be rushed. However, given ONS has missed the dates for committed publications, it is reasonable for users to expect a realistic roadmap for the research that maintains the momentum of its original ambition for the research.

2.33 Given the volume of work ONS has carried out to investigate these data further, ONS must bring users along with it as its understanding of the data improves. The external communication around the work published so far and the delays to date could give the impression to users that ONS is not fully explaining the potential issues with the data on gender identity. Media coverage suggests these statements may have reinforced the perception of quality of the published estimates. However, in the absence of ONS’s full research outputs, they present only a partial picture. This has the potential to undermine confidence in the good analysis that has been carried out.

2.34 On the current gender identity data harmonised standard page, ONS has added a box which states “The Census 2021 gender identity question is the question that tested best with respondents prior to the census being conducted in March 2021 and remains the current recommended standard. Given that this is the first time that these data have been collected, the standard’s current status is “under development”. The banner also advises those considering using the question to contact ONS for support while it carries out further research into the quality of gender identity data. We welcome the clarity of this message and the support that is offered to those considering using the question. It would be beneficial to users for ONS to include this wording or something similar within the published quality information that is signposted from the gender identity datasets for transparency.