Ed Humpherson to Tony Dent: Better Statistics CIC

Dear Tony

Thank you for your letter of 28 October. I am pleased to hear about the positive engagement that Better Statistics has had with Savanta and the Centre for Data Ethics and Innovation (CDEI) on the CDEI tracker survey. There are four further issues you raise which I would like to address in this letter: the Office for National Statistics (ONS) COVID-19 Infection Survey (CIS) and our recent review of it; communicating uncertainty; our review of population estimates; and the Code of Practice for Statistics.

Before taking these issues in turn, I wanted to reassure you that, even where we do not agree, your questions, observations and challenges are useful for us – and we do in fact agree with you on several points as you will see below. However, I do feel that our engagement would be enhanced through face-to-face conversations. With that in mind, I would like to suggest that we meet to discuss how we work with you and your Better Statistics colleagues, and how we can engage in open discussions about the UK’s statistical system.

Office for National Statistics COVID-19 Infection Survey

The main purpose of our review of the CIS was to consider whether, and to what extent, the statistics continued to meet the standards of the Code of Practice for Statistics in light of plans to scale the survey back. Our review gathered a range of evidence, including documentation from ONS, interviews with ONS and feedback from users of the statistics. We concluded that the statistics from the CIS continue to be of sufficient quality and fit for their intended uses. This conclusion was based on a combination of factors, such as user needs, use of appropriate methods, thorough quality assurance processes, efforts taken to improve representativeness and the involvement of experts in various fields.

While the conclusion in this review remains our current position on the CIS, I take very seriously the concerns you raise both about the quality of the statistics and our review of them. As such, I am in the process of commissioning an external expert to provide an independent assessment on both issues. We will keep you informed about the independent assessment, and of course publish the outcome of this review in due course.

We have also continued to engage with ONS on the more specific points you raise regarding the CIS. I have provided an update on these below:

We consider that the Methods article published on 5 August gives a good account of methods and quality information for the CIS and provided a fuller explanation of some of the points that you had raised with us previously, such as whether household size is included in all analyses and the reason in some cases that it is not. ONS has informed us that an updated Methods article following the move to digital data collection will be published in the coming weeks.
ONS told us that it remains committed to publishing more information on response rates. ONS has been carrying out this analysis and hopes to publish it soon.
You raise a further point that ONS should consider alternative methods to the MRP model which may be able to better account for household size. The modelling approach taken by ONS is considered by many statisticians to be a state-of-the-art technique and appropriate for use in this context. While it’s possible that other models could have been used, we consider that this method is appropriate for the survey. However, this is an issue which we will ask an external expert to consider as part of the independent review.
Finally, with regards to value for money of the survey, you mention a request to investigate the clauses of the contract between ONS and its survey provider IQVIA. We are not aware of having received such a request. In any case, examining a contract would be beyond our remit and more relevant for financial auditors. Separately, we understand that although there was previously some confusion on ONS’s behalf about documentation which you had requested from IQVIA, this will be shared with you imminently.

Communicating uncertainty

I agree with you that it is essential that uncertainty around statistics is communicated well to support appropriate use and interpretation. I also agree that the fortnightly ONS business survey (BICS) is an example of statistical reporting which does not adequately convey the uncertainty around the estimates presented. My team has raised this with ONS and asked it to make improvements to the publication at the soonest opportunity. We will review these statistics as part of a planned compliance check in the last quarter of the 2022/23 financial year. More broadly, we continue to support and challenge the statistical system to improve how it communicates uncertainty, as we follow up our recent report on this topic.

OSR’s review of population estimates

The purpose of our review was to consider whether ONS’s estimates and projections could bear the weight that is put on them, reviewing the methods and data that underpin these statistics in line with the Code. The review was broad in its nature, as was our intended audience with such a range of users of these data. As part of our review, we spoke with various stakeholders including other government departments, representatives from several different local areas, and expert demographers and population specialists in academia. While our intention was not to focus only on issues experienced by users in areas such as Coventry, we did consider user perspectives during our review. Indeed, one of the conclusions of our review was that ONS did not adequately consider the concerns raised by some users. As you suggest, this may be an issue where we will need to agree to disagree.

Code of Practice for Statistics

As set out in my letter to you of 28 April, we do not make judgments on value for money of government spending. Nevertheless, we consider that the Code sufficiently addresses key aspects of the effective use of data and other resources in relation to statistical practice.

However, we always welcome challenge to our work and to potential improvements to the Code, and we like your suggestion of a more detailed discussion about the Code. To that end, we propose that we should convene an open discussion with stakeholders, including Better Statistics, next year to examine the purpose of the Code, how it helps and where it could help more.

If you would like to discuss any of these issues, please let me know and my team will be happy to arrange a meeting.

Yours sincerely

Ed Humpherson
Director General for Regulation

Ed Humpherson to Tony Dent: Better Statistics CIC

Office for National Statistics COVID-19 Infection Survey

Communicating uncertainty

OSR’s review of population estimates

Code of Practice for Statistics

Related links