Ed Humpherson to Iain Bell: ONS COVID-19 Infection Survey ad hoc teacher analysis

Dear Iain,

Since we wrote our letter to Dr Rasmussen about the ONS COVID-19 Infection Survey ad hoc teacher analysis, we have done further work on the issues she raised.

We recognise the challenge in clearly communicating technical information to users, particularly in a context where you are striving to turn around important analysis very quickly. However, it is important to clearly explain findings such as ‘no evidence of a difference’ to avoid misinterpretation by others.

We consider that after concerns were raised about the initial publication of the ad hoc teacher analysis, ONS should have elaborated on the strengths and limitations of the analysis, what conclusions can be drawn and what these conclusions mean – for example, in relation to the impact of the rate of prevalence on the findings and the ability of the survey to determine differences between groups.

Given the broad range of users and occupations with an interest in data from the Infection Survey, we think it is fair that you are now prioritising a more comprehensive analysis of occupational risk. We have outlined below our advice regarding the development and communication of this and similar analysis in future:

  • As discussed with the CIS analysis team, we encourage ONS to be open about its plans for upcoming ad hoc analysis topics. Sharing these plans as quickly and as widely as possible would provide ONS with an opportunity to engage and collaborate with expert users to improve the final analysis.
  • We reiterate our previous view that ONS should be clear with users about the ability of the survey to detect differences between groups, particularly for analyses with small numbers of positive cases. ONS must clearly communicate its findings, including appropriate commentary and caveats.
  • ONS should explain its decisions on methods choices to users. For example, in relation to the new occupational risk analysis, what are the criteria for an occupation being classed as high risk? And what is the reason for a two-stage approach to the analysis rather than analysing the risk of infection and death for all occupations?
  • ONS should offer as much insight as possible into how interventions that were introduced for some workers at different stages of the pandemic impact the findings. It should also ensure that findings from this analysis are presented in the context of previous findings about differences in infection rates between occupations.

I would like to thank you and your team for your ongoing engagement with us as we carry out our review of the survey. We are happy to provide further support and advice as you continue to develop these statistics.

Yours sincerely

Ed Humpherson

Director General for Regulation

Mark Pont to Rob Kent-Smith: Experimental Estimates of GVA using Double Deflation

Dear Rob

Experimental Estimates of GVA (Gross Value Added) using Double Deflation

I am writing to you following our review of ONS’s recent publication: Producing an alternate approach to GDP using experimental estimates of GVA. These new experimental estimates of GVA represent a significant step forward towards improving the UK’s National Accounts framework. Implementing double deflation should minimise industry-level bias in GVA statistics and, hence, make them more accurate and valuable. We decided to conduct this review in order to evaluate progress in the development of these estimates and to identify where users can expect further enhancement of the statistics going forward.

Producing volume estimates of GVA using a double deflation method is widely regarded as international best practise for calculating estimates of real-terms Gross Domestic Product (GDP(P)) and has been a key recommendation of several reviews of economic statistics in recent years.

We welcome the positive progress that has been made towards developing a double deflation approach to National Accounting and would particularly highlight the following:

  • There is a high degree of transparency regarding both the methodology and the development plans for these statistics. ONS has also published several iterations of a deflator strategy over the last few years, which further enhances user understanding of the way the estimates are produced
  • The background and context to the statistics are explained at a level that is appropriate for the kinds of people who will be engaging with these statistics, who will be largely experienced, technical users
  • The publication is clear about the experimental nature of these statistics and their current limitations
  • There has been good engagement with user groups throughout the development of these statistics. Efforts to produce an internal ‘deflator dashboard’ and share this with some users, in order to facilitate understanding of deflator strategy, are particularly commendable. Engaging with users in this way is likely to increase their understanding of the methods behind these estimates, and result in more meaningful conversations between producers and users going forward

As you continue to develop these statistics, the following points will be amongst the most important to consider:

  1. Continuing work alongside the ONS team producing Services Producer Price Indices (SPPIs), to identify what changes to SPPIs are required in order to make them suitable for use in the National Accounts framework, whilst also allowing them to remain valuable price indices in their own right. SPPIs can be a more accurate deflator of intermediate consumption than industry deflators, as they are likely to be more reflective of the price paid by industry. Currently 49 of the approximately 200 SPPIs are being used in the National Accounts and consideration should continue to be given as to which price indices will provide the best possible deflator for the respective consumption.
  2. Consider how best to communicate to users the greater volatility associated with double deflated estimates of GVA in comparison to figures produced using single deflation. We are pleased to hear that you are already endeavouring to understand the volatility of these estimates by engaging with international comparators, and would encourage you to share these findings with users once these have been gathered
  3. Consider how best to alert users of instances where a price index and a deflator differ. We understand that this will become a more common occurrence in the future, and it is important to ensure that users understand why this can happen and the resulting consequences for the statistics they use
  4. Consider how best to communicate the potential impacts of double deflation on labour productivity before the new approach is fully adopted into the UK National Accounts. The new framework for volume estimates of GVA should produce better estimates of productivity analysis at the industry level. Given the interest in the UK’s productivity puzzle amongst users of productivity statistics, there is likely to be considerable interest in the potential impacts on UK productivity from the implementation of the new framework for double deflation and users should be made clear about the implications of these.

I’d like to thank you and your team for your positive engagement on this review. We have been very pleased to see the progress you have made with these double-deflated estimates, and look forward to seeing the further developments made going forward. I am copying this letter to Darren Morgan, Director of Economics Statistics Development at ONS.

Yours sincerely

 

Mark Pont

Assessment Programme Lead

Ed Humpherson to Iain Bell: Communicating models and methods to users of the Coronavirus (COVID-19) Infection Survey

Dear Iain

Communicating models and methods to users of the Coronavirus (COVID-19) Infection Survey

Thank you to you and your team for engaging with us as we investigated concerns about communication of the methods used to produce the estimates of the incidence rate of COVID-19 from the Coronavirus (COVID-19) Infection Survey. These have centred around the availability of clear information about methods and quality that meets the needs of all types of users, and developing an ongoing positive dialogue with users.

Estimates of COVID-19 positivity and incidence rates are widely used and play a vital role in informing the government response to the pandemic. These statistics are unusual in their complexity, and their wide user base, so it is important that the modelling and data are presented in a clear and accessible way to this broad range of users to meet their needs. The challenge is that the needs of general users must be met, by making the explanations accessible, and at the same time, the needs of technical users must be met, by providing adequate detail about the methods used.

The additional information published in the bulletin on 11 December regarding the calculation of incidence rates demonstrates ONS’s commitment to improving its communication of the underlying methods. However, it is clear that the presentation of the statistics and descriptions of the methods needs to be much clearer still, both for general and technical users. For example, the distinction between the ‘official reported estimate’ and ‘modelled estimate’ of the incidence rate remains a source of confusion for users, particularly as the estimates are presented alongside each other.

Due to the complexity of the statistical models, it is also important that detailed information is accessible to technical users to allow them to evaluate the methods used and to fully understand the data and the way that estimates change due to the modelling. The pre-print academic paper (which has yet to be peer-reviewed), submitted by your academic partners, goes into considerable detail on the methods used. However, it would be helpful to reflect upon recent concerns raised by users and work with users to produce clear and accessible summaries of the rationale for the methodology choices made. This should include information about the assumptions made and how the performance of the models is evaluated and improved.

The COVID-19 Infection Survey continues to deliver valuable insight about the pandemic, at speed and to a wide range of users, and we support its ongoing development. We appreciate the resource constraints and pressure that you are under, but it would benefit ONS to demonstrate more clearly that it is open to engagement with users. This would be achieved by sharing information about the user engagement you are already undertaking and engaging more openly with a wide range of users, including those with new requests or questions.

We will follow up the points raised in this letter through our in-depth review of the survey.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

Ed Humpherson to Iain Bell: COVID-19 infection survey publication time

Dear Iain 

COVID-19 infection survey publication time 

Following conversations with your team, I am happy to confirm the one-off exemption from your usual publication time for COVID-19 Infection Survey statistics of noon on a Friday, to 10:30am on Thursday 26 November.  

I very much welcome your commitment to review your publication schedule to enable you to respond swiftly to emerging policy decisions around COVID-19.  

Yours sincerely  

 

Ed Humpherson 

Director General for Regulation 

Mark Pont to Rob Pike: Mergers and Acquisitions Statistics (ONS)

Dear Rob

Mergers and Acquisitions Statistics

Thank you for your commitment to developing the public value of ONS’s Mergers and Acquisitions statistics and your team’s participation in our recent review of them. This letter confirms our findings and the continued designation of these statistics as National Statistics.

We chose to review these statistics given their increasing importance in gauging the impact of Brexit on the UK’s investment landscape, and to assess the impact of the change in data collection method in 2018 on their quality and public value.

Overall, we found the statistical release and Mergers and Acquisitions data to be of good quality and to have improved since the last assessment in 2011. Our review found a range of positive features, which demonstrate the trustworthiness, quality and the value of the statistics, including:

  • the extensive work undertaken by your team to develop these statistics by increasing the scope of data capture and reducing the reliance on surveys via the use of the Bureau Van Dijk (BVD) Zephyr database.
  • combining Mergers and Acquisitions vendor information, with statutory survey data for deals over £100m, which benefits users by increasing coverage and providing more timely estimates of transactions.
  • the introduction of innovations, such as publishing completed deal information in the quarterly statistical bulletins, to aid the analysis of COVID-19 impacts on Mergers and Acquisitions activity. In addition to breakdowns by continent, these developments have helped satisfy users’ requests.
  • the additional work ONS has undertaken to update the quality and methodology information, by publishing details of the thorough quality assurance work carried out following data collection and ongoing reviews of data sources. We consider it good practice that users can interrogate the results and explore anomalies or interesting findings at stakeholder meetings.
  • the improved commentary your team has added on the impacts of EU withdrawal, including the use of evidence from the Bank of England’s Credit Conditions Survey and Agents’ Summary of Business Conditions report, all of which aid user interpretation of these statistics.
  • the ongoing discussions ONS is having on publishing inward and outward microdata at the minority ownership level (10%). This innovation will help improve continuity between Mergers and Acquisitions data and Foreign Direct Investment data.

We have identified some areas where we consider that the statistics should be enhanced in order to ensure that the highest standards of the Code of Practice continue to be met: ONS should seek to:

  • engage with users on the impact of recent developments to the statistical bulletin, improvements in coverage and the new imputation methods for missing deal values.
  • evaluate from user feedback whether there is a role for data (and supporting narrative) on the announcement of proposed deals to be used as a leading indicator of economic activity.
  • identify opportunities for linking announcement and completion date data with other business survey data, to increase the evidence base available to policymakers designing and managing macroeconomic, industrial and economic development strategies.
  • update the Quality and Methodology information so that it includes advice on how to compare estimates from the Zephyr database with historic data and narrative on how quarterly outputs of Mergers and Acquisitions deals are constructed.

Thank you again for the commitment and enthusiasm you and the team have shown in already starting to address the enhancements we have outlined. I am copying this letter to Matt Hughes, Assistant Deputy Director Global Trade & Investment; Lee Mallett, Economic Advisor, International Investments Branch; and Gill Sanderson, Statistical Results and Publication Manager, Mergers and Acquisitions Surveys.

Yours sincerely

Mark Pont
Assessment Programme Lead

 

Mark Pont to Mike Hardie: Consumer Price Inflation including Owner-Occupied Housing Costs (CPIH)

Dear Mike

Consumer Prices Index Including Owner-Occupiers’ Housing Costs (CPIH)

Thank you for the positive engagement of yourself and colleagues during our follow-up review of ONS’s development of CPIH – Consumer Prices Index including Owner-Occupiers’ Housing Costs (OOH). This review follows our letter from July 2017, when the UK Statistics Authority re-designated CPIH as National Statistics. We focused on the two key areas from our previous letter: providing analyses about OOH, and continuing to build effective user engagement.

Your ongoing publication of comparative analyses continues to provide reassurance about various aspects of the CPIH. We particularly welcome that since you have gained access to record-level data from the Valuation Office Agency, you are reviewing alternative ways to estimate OOH. In addition, we see ONS’s innovation in integrating web-scraped prices and those gleaned from scanner data as leading-edge developments in measuring inflation resulting in significant improvements to the methods and accuracy of the CPIH statistics. Also, these new methods should lead to improved product coverage, higher frequency of collection and potential cost savings.

We commend the steps that ONS has taken to enhance engagement with users, in particular your engagement with the Stakeholder and Technical Advisory Panels on Consumer Prices. Setting out the landscape of inflation statistics through the use cases will continue to be important particularly in the light of proposed changes to the Retail Prices Index (RPI) and the development of Household Costs Indices (HCIs). We recognise your commitment to keep the use cases updated.

We have expanded on the above points in an attached annex. I am copying this letter to Jonathan Athow, Deputy National Statistician for Economic Statistics.

Yours sincerely

Mark Pont

Assessment Programme Lead

 

Annex: This annex sets out the main findings of our review

Related Links

Ed Humpherson to John Pullinger: Consumer Prices Index including Owner Occupiers’ Housing Costs (CPIH) re-designation

Mark Pont to Ole Black: Statistics on Internet Access and Use

Dear Ole

Thank you to your team for engaging with us as we discussed the findings of our recent review of ONS’s internet-related statistics, published in two streams as: Internet access – households and individuals, Great Britain and Internet Users, UK.

We recognise the importance of having high quality statistics about internet access and use which serve the public good. We welcome the methodological improvements made relating to the Internet Access statistics with the transformation from face-to-face to a mixed-mode design. It is good that this generated a sufficiently large returned sample that the statistics for 2020 were robust enough not to be affected by COVID-19.

Our review found that ONS’s internet statistics do not cover key topics such as the speed and equality of broadband access, and provision of fibre broadband. In response to our review and the challenges presented by the COVID-19 pandemic, we welcome your commitment to pause and review your stakeholder requirements for statistics on the digital economy, including access and use of the internet. This provides a great opportunity to work collaboratively with others, and look to harness the power of alternative data sources to deliver greater insights. We look forward to ONS engaging and working with stakeholders to develop these statistics.

I have copied this letter to Phil Davies in your team.

Yours sincerely

 

Mark Pont

Assessment Programme Lead

Ed Humpherson to Sir Ian Diamond: Assessment of Business Demography Statistics

Dear Sir Ian

Assessment of Business Demography Statistics

We have completed our assessment of ONS’s business demography statistics. I am grateful for the positive contribution from you and your team throughout the assessment.

Our report suggests that a range of improvements is needed for these statistics to fully meet National Statistics standards. We will formally review whether the statistics merit their continued designation as National Statistics at two key points.

First, upon publication of the next set of annual business demography statistics, we expect ONS to have made some short-term gains. Second, ONS must publish a detailed plan, explaining how it will address the improvements identified in this report, including plans for reviewing the funding of the Statistical Business Register.

Business demography statistics are crucial for understanding UK business dynamics, especially as the economy emerges from the COVID-19 pandemic. I am impressed with ONS’s innovative work to develop, in partnership with Companies House, a weekly indicator of business births and deaths to track the impact of the COVID-19 pandemic. The subsequent publication of experimental quarterly business demography statistics has further demonstrated ONS’s ability to make a sizeable shift in its data offering. In making the improvements needed, I hope that ONS can build on its work over the last few months.

ONS has an amazing, yet largely untapped, resource in the Inter-Departmental Business Register (IDBR) that with development could help business demography statistics achieve the status of leading economic indicators, as they have in the USA and France. Part of untapping the value of IDBR data is making them available, subject to appropriate safeguards around confidentiality. We note that the legislation under which much IDBR data are collected and held – the Statistics of Trade Act – is over 70 years old. We would support any efforts that ONS were to make to review this legislation.

I note that organisational responsibility for the IDBR is separate from the rest of economic statistics. This has resulted in the IDBR not receiving the corporate priority that it clearly needs, nor playing as integral a role in economic statistics as it should. In responding to the requirements in our assessment, I encourage ONS to review these responsibilities and ensure that the IDBR can sit at the heart of economic statistics production.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

 

Related Links

Assessment Report: UK Business Demography Statistics

Ed Humpherson to Iain Bell: Compliance check of ONS Overseas Travel and Tourism statistics

Dear Iain

Compliance check of ONS Overseas Travel and Tourism statistics

I am writing about the compliance check of ONS Overseas Travel and Tourism statistics which we carried out last year. Our review focused on two issues: the sample size of the survey used to calculate the statistics and how the tourism team engaged with users. We set out several requirements that ONS was to meet by July 2020 to ensure compliance with the Code of Practice for Statistics. These included improvements to user engagement and publishing better information about quality and methods.

I would like to commend the hard work your team has carried out since our review. Taking into account the impact of COVID-19 on the International Passenger Survey, which provides the data for these statistics, we consider that ONS has demonstrated great progress in meeting our requirements. Developing new ways to engage with users has resulted in ONS reaching and understanding a broader group of users – for example hosting user events and seminars, carrying out an online survey and distributing a regular newsletter. Implementing an improved methodology to deal with the known underestimates for visitors from some countries means that users have more accurate information. Users are also now better supported in understanding the statistics and their limitations thanks to your published guidance on interpreting confidence intervals. I welcome your commitment to ensure the longevity and sustainability of these improvements.

I would also like to welcome your review of tourism statistics which you plan to carry out over the next six months and support your plans to explore additional sources to further enhance the scope and value of these statistics. I encourage you to ensure that the approach to, and findings of, the review are open and transparent to users. Given that COVID-19 led to suspension of International Passenger Survey operations in mid-March, I welcome your team’s efforts to publish tourism statistics for the affected period. This approach meets user needs as far as possible, while acknowledging the impact on quality through temporary suspension of National Statistics status.

We appreciate the openness and enthusiasm with which your team has engaged with us over the last year and look forward to receiving further updates on the outcome of your review.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

Ed Humpherson to Iain Bell: Assessment of 2021 Censuses in the UK – ONS Response to Preliminary Findings

Dear Iain

Assessment of 2021 Censuses in the UK – Office for National Statistics (ONS) Response to Preliminary Findings

We have reviewed the actions that your team has taken to address the findings in Assessment Report 345: 2021 Censuses in the UK – Preliminary Findings. These findings aim to build on the existing work being undertaken by Census offices, providing further direction and focus on pre-existing plans.

The Office for Statistics Regulation’s (OSR’s) advice on the National Statistics designation will be based on the trustworthiness, quality and value secured by Census offices. That advice will take account of the actions taken to address findings or requirements from OSR as part of this assessment process. While many of the Phase 1 findings can only be fully actioned or addressed over longer timescales, it is my view that ONS has responded positively to the assessment findings so far. ONS has been open to challenge from the assessment team and I am encouraged that improvements have been made in a number of areas. The Census is one of the most important sources of official statistics and I recognise the ongoing commitment of ONS to deliver high quality data and statistics; recognising the challenges and changes in ways of working that have had to be managed during the COVID-19 pandemic.

Since we published our assessment report in October 2019, my team has engaged with various members of your staff working on the Census to understand the improvements you have made since then. I welcome the openness and flexibility that has been shown to the assessment team. In evaluating how ONS has responded to our findings we have taken account of published materials from ONS including the evidence report Response to actionable findings from Phase 1 of the National Statistics Accreditation, alongside information gathered through meetings and workshops held with your teams. I consider that some actions taken by ONS are of note.

  • My team held a number of workshops with staff who have been working to develop methods, on areas such as item imputation and census coverage adjustment. We consider that ONS is taking a rigorous approach to its methods development. We are very pleased to see that methods papers, which have been reviewed by your external Methodological Assurance Review panel, are now in the public domain. This is a significant improvement to aid transparency and share knowledge and expertise with the wider community of expert users and stakeholders.
  • Some findings from our assessment report involved making improvements to recognise the needs of UK Census data users. ONS has taken a lead role in addressing these findings and has established a UK Census data working group – with representation from across the Census offices; we recognise the commitment and energy of ONS staff working in this area.
  • The questions to be asked in the Census in 2021 have been finalised. However, we are aware that some stakeholders have outstanding concerns with the guidance on the Census sex question. ONS has continued to engage with users and stakeholders about this and we recognise the efforts being made to provide answers or explanations on areas of concern. ONS should seek to address outstanding concerns raised by users within its further question testing and research on the guidance on the sex question. ONS should share the outcomes of this research in a transparent and open way.

We consider that further action is still needed in some areas identified for improvement in our assessment report – this is partly due to changing circumstances such as the impacts on programme delivery given the pandemic. For example, while ONS has plans to make improvements to its website, to date there has been little change made to make it more accessible. Key documents and materials are not easy to find and navigation on the webpages is not intuitive. ONS should implement its plans over the coming months to improve the accessibility of research and other Census information on its website.

In the assessment report the need to be clear to users on the impacts of country-specific decision making on UK Census data and statistics. Due to the decision by Scottish Ministers to delay the Census in Scotland, it is even more important ONS work with the other Census offices to explain the impacts of this to UK Census data users.

From speaking with your teams, it is clear ONS recognises the importance of communicating to England and Wales Census data users about delivering its Census in 2021 and on its scheduled plans for Census outputs. Users should be kept up to date and be provided with further detail as more information on the timing and content of statistical outputs becomes available.

We have included more detail about our judgement in an annex to this letter (See PDF download). I, or my team, would be happy to talk you or your colleagues through any aspects of this letter or Code compliance more generally.

I am copying this letter to Sir Ian Diamond, National Statistician and Stephanie Howarth, Interim Chief Statistician, Welsh Government.

Yours sincerely

Ed Humpherson
Director General for Regulation