OSR’s regulatory decision

This section sets out our regulatory decision, based on the timelines set out in the first section of the report, and the analytical concerns in the second section.

As noted in our interim report, a user need for data on gender identity was identified as part of ONS’s public consultation on proposed topics for the 2021 Census questionnaire in England and Wales. In particular, the ability to capture data on the trans population was seen as a data gap for service provision that was important to address.

To help establish the extent to which the census gender identity data are being used, we asked all the stakeholders we talked to if they were aware of any use cases of the data. Some stakeholders stated that they believed the data were being used to support decision making, but in these cases, they were likely not the only source of evidence being drawn upon.

Several local authorities, such as Shropshire Council and West Yorkshire Combined Authority, have published dashboards, data tables or reports that summarise or map the trans population within their area. Similar local analysis has been undertaken by the House of Commons Library and some third-sector organisations, such as Trust for London, which produced an interactive map exploring the link between gender identity and deprivation. In the course of this review, we were not able to find examples of these census data being directly cited in a policy decision or action in support of resource allocation.

We acknowledge that ONS’s request to label the census gender identity data as statistics in development supports its desire to publicly reflect the evolving understanding of gender identity. It also shows that ONS recognises it can do more to support users in interpreting the census gender identity data and understanding any limitations.

To determine our regulatory decision, we have considered whether the census gender identity data meet the standards set out in the Code of Practice for Statistics. In particular, we have looked at how well the ONS research report published in November 2023 addressed concerns about potential misunderstandings of the gender identity question and how such issues might have affected the quality of the gender identity data in the census. We have also formed a view on whether the publication of the research report has mitigated any remaining concerns by providing increased quality information on uncertainty or guidance on using the statistics. Through our review, we have considered three key areas of the Code:

Quality – Suitable data sources

As ONS has now recognised, the question developed to determine the size of the trans population of England and Wales did not work as intended. The evidence indicates that people may have found the question confusing and therefore gave a response that did not reflect their gender identity. This appears to be more likely for people who do not speak English as their first language. The different question adopted for the Scottish Census does not seem to have encountered the same difficulties.

It is important to note that neither ONS, nor we at OSR, identified in the development or delivery process of the census that there was a significant risk that certain groups of people were more likely than others to misunderstand the question.

Assured Quality

A core requirement of the Code of Practice is that statistics producers conduct suitable proportional quality assurance, and that they are transparent about the quality assurance approach taken throughout the preparation of the statistics.

There are a number of areas where ONS could have better communicated with users of the census gender identity statistics. Our interim report found that ONS’s communications were not sufficiently managing users’ expectations on what areas ONS’s research was exploring and what the research might show. Upon publishing the research, which was preannounced, delayed twice and cut from four outputs to one, ONS made no acknowledgment of any changes to outputs. This approach had the potential to damage trust and confidence in the research.

Additionally, until recently, ONS’s approach has been characterised by a focus on defending the estimates of the size of the trans population. ONS has not shown sufficient willingness to engage constructively with the emergent evidence. The publication of the Scotland Census data on trans and trans history in June 2024 has been instrumental in a shift in ONS’s approach.

We have confidence that those working on the data and research have been working in line with the Code, but consider that stronger leadership from ONS, particularly on clearer public messaging when concerns were raised and on setting expected outcomes for the work, was needed.

Communicating uncertainty

The Code requires that the extent and nature of any uncertainty in the estimates be clearly explained. ONS has conducted a range of research activities to explore the quality of the estimates, but these efforts have been hampered by a reluctance to communicate the limitations of the gender identity statistics. ONS’s headline message in the research report around the validity of the census data, and how users should interpret and use the data at national and smaller area levels, is unclear.

Our interim report found that ONS should have communicated the inherent uncertainties in this new data collection to aid use when it published the data. We noted that had ONS done so, it would have aided users’ understanding of the appropriate use of the data and provided relevant context when issues were identified by users. In addition, we stated that ONS could have been more open and transparent about these novel data and the work that has followed and that ONS’s planned research must communicate how the data can and cannot be used. We have found that the research report does not fully meet this need and ONS has not done enough to communicate uncertainty in line with the expectation.

Users who use the ONS census outputs at a national level would struggle to locate any of this research to understand the impact of the research on the use of the gender identity data. This problem is replicated at the local level, where the research indicates that the variability of the estimate may be greater.

We have considered three further factors relating to the status of these estimates:

ONS’s approach to criticism of the gender identity data

We noted in our interim report that the content and tone of the published communication from ONS has been somewhat closed and at times defensive. Some users have raised concerns that ONS has been captured by interest groups, leading to a lack of objectivity. We found no evidence of this form of bias through any of our work. We consider it regrettable that ONS’s defensiveness has created an impression of bias for some external observers.

Whether the statistics are materially misleading

The purpose of the Code’s Quality pillar is to ensure that statistics are a fair representation of what they seek to measure and are not materially misleading.

ONS confirmed in its letter to us that it is unable to quantify all the uncertainty surrounding responses to the Census 2021 question on gender identity, and that the available evidence does show there is potential for bias in how the question was answered by those who responded that they did not speak English well. The evidence we have considered in this review leads us to the conclusion that considering the understanding of the question alone, this issue is highly likely to have led to an overestimate.

However, other sources of potential bias (for example non-response) in what was a new data collection are not fully known. The ONS research report is correct to say that it “cannot say with certainty whether the census estimates are more likely to be an overestimate or an underestimate of the total number of trans people aged over 16 years in England and Wales.”

At the national level, triangulation with other sources, including data from the Scottish Census, suggest that an estimate of around “1 in 200” triangulates with other sources and is not likely to be materially misleading. While the information on the size and nature of all the potential biases is incomplete, it is hard to draw the same conclusion for some more-detailed breakdowns, including for local areas where the data indicate a higher concentration of people misunderstanding the question.

Our regulatory approach

It would be reasonable for users to expect that a new question with few data available for triangulation would have extra information provided to support appropriate interpretation and use.

The Code of Practice places a strong emphasises on innovation and improvement. It recognises that the world does not remain static, and that change is an inevitable part of life. We encourage statistics producers to identify opportunities to use new sources and methods and to recognise any fresh needs for information among users of statistics.

When developing new statistics or introducing a new data source or methodology, initial publications are often labelled ‘official statistics in development’ by producers. We would usually assess the official statistics against the Code of Practice once the development is complete.

However, the census is a unique case. Our regulatory accreditation process was undertaken in tandem with the development and first publication of the statistics in the respective countries of the UK. While this process allows us to provide users of these statistics with the reassurance of quality that comes with accreditation at first publication, it means that if we have any subsequent concerns about any aspect of the Code of Practice for Statistics, we may need to reconsider our accreditation decision in light of new evidence.

In this case, new evidence has emerged that has demonstrated that the statistics do not comply with important quality aspects of the Code of Practice for Statistics. With the benefit of hindsight and this new evidence, our conclusion is that the census gender identity statistics, which were collected from a novel question in an area where measurement practices are still emerging, would be better described as official statistics in development, and should not be labelled as accredited official statistics.

We acknowledge that this is a limitation with our approach, and as a result of this review, we are reviewing what changes might be needed to our assessment approach in circumstances where accreditation at first publication is in line with serving the public good.

Our decision

The correlational evidence set out in this report supports the view that some groups of people were more likely than others to misunderstand the question. This view is also supported by ONS’s finding that there are patterns in the data that are consistent with some respondents not interpreting the question as had been intended.

Though the additional data tables published alongside the Quality of Census 2021 gender identity data report are useful for understanding the nature of responses and their relationships with other characteristics, we consider that the research report and data tables have not fully addressed the concerns raised by users.

Given our findings, we have written to ONS to support its proposal to publish the Census 2021 gender identity statistics in England and Wales as official statistics in development rather than accredited official statistics. We found that the issues highlighted in this report are unique to the statistics on gender identity and therefore all other outputs from the Census 2021 in England and Wales are unaffected by this decision and remain designated as accredited official statistics (called National Statistics in the Statistics and Registration Service Act 2007).