Executive Summary
Spotlight on Quality: Assuring Confidence in Economic Statistics
ES.1 As a result of the UK’s departure from the EU, Eurostat no longer provides external assurance of the UK’s economic statistics. We recognise that key stakeholders, and the wider public, need additional assurance, specifically on the quality and independence of economic statistics.
ES.2 We are now delivering a series of quality-focused assessments to provide this enhanced assurance, using a newly developed assessment framework that focuses more intensively on the quality of economic statistics. The assessment framework will be developed further as the Spotlight on Quality: Assuring Confidence in Economic Statistics programme progresses.
What we found
ES.3 We have identified eight actions for ONS to improve the quality of the Producer Price Indices (PPI). Implementing these actions will ensure that the statistics continue to meet the highest standards of the Code of Practice for Statistics. We expect ONS to publish an action plan by September 2023, setting out how it intends to improve the quality of the PPIs, and report back to us every six months on progress on implementing the actions.
ES.4 ONS has made some major improvements to the quality and international comparability of the PPIs in recent years, mainly through method changes that have brought the methods in line with international best practice and the needs of users. The most significant change is the implementation of annual rebasing and chain-linking, which ensures that PPIs are better equipped to adapt to structural changes in the UK economy, while enabling comparison of price changes over time. ONS has also changed the headline input index to a gross measure. It used to also publish an alternative measure, compiled using the net of inter-sector transaction weights, but this was discontinued. The single headline input index on a gross measure reflects international best practice and user needs.
ES.5 However, to deliver these improvements, ONS has paused work on updating the samples of the three statutory surveys used to collect price data. Production of PPIs also relies on an inflexible legacy system, the Ingres relational database, which poses risks to the quality of the statistics.
ES.6 ONS’s current statistics prioritisation framework prioritises more-prominent, market-sensitive economic statistics such as consumer price inflation over statistics with a lower profile and smaller user base such as PPIs. While it is right that ONS prioritises the more-important statistics, years of under-prioritisation have negatively affected the quality of the PPIs.
ES.7 Without reprioritisation and investment, there is a risk that the quality of the PPIs will further decline and that they become less robust, particularly in their primary use as a deflator for other economic series. It is good that ONS is now formalising development plans to address some of the main quality issues with the statistics, including the survey samples and legacy system.
ES.8 The price statistics experts from the international statistical community that we spoke to praised ONS’s methods and agreed that ONS follows international best practice in producing PPIs. The business prices team told us that it intends to remain consistent with international best practice as much as possible for producing PPIs but will examine where it may be appropriate to depart from international best practice to best meet UK users’ needs.
ES.9 ONS is not achieving the target number of price observations for the three surveys used to collect price data, and in some cases is now unable to provide a reliable deflator for certain industries. Updates to the sample were paused in 2019, to implement the above methods changes, and this has affected the representativeness of the samples. Not updating the sample has also exacerbated the impact of sample attrition, which has increased substantially in recent years.
ES.10 As with other ONS business surveys, the COVID-19 pandemic significantly affected the response rates of the three surveys. The falling response rates mean that the business prices team has less data to compile the indices from. As a result, the team is carrying out far more imputation than it used to before the pandemic, which it told us can lead to price changes that are less representative of transactions.
ES.11 We see these survey issues as the biggest threat to the quality of the statistics. It is essential that ONS reviews and updates the survey samples to improve the quality of the PPIs (Requirement 1). Additionally, ONS should review its imputation methods to ensure that they are fit for purpose and not introducing bias (Requirement 6).
ES.12 ONS is currently developing an electronic questionnaire for the three surveys and expects to roll out this long-awaited improvement at the end of 2024 or early 2025. We welcome the move to online data collection. The use of electronic data collection methods will support higher quality data by making data easier to return for manufacturers and enabling point of collection validation.
ES.13 The legacy Ingres-based system is inflexible, vulnerable to processing errors, and has hindered significant development of certain aspects of the production process. It is good that ONS is planning to migrate all existing business surveys to a new, more flexible platform, and is implementing an Ingres-reduction strategy in the meantime. Replacing the Ingres-based system is essential for futureproofing the production and development of PPIs (Requirement 2).
ES.14 The quality assurance (QA) process for the PPIs is well-established. However, a series of errors identified between November and December 2022 highlighted gaps in the QA process. The business prices team used this an opportunity to strengthen its QA process: it asked ONS’s Methodology and Quality Division (MQD) to carry out a review of the process, to provide an additional layer of assurance. The improvements implemented as a result of the review have reduced the risk of future errors, and we welcome the team’s openness to peer review. ONS could further strengthen its QA process by implementing the outstanding recommendations of the MQD review. Users told us that they welcomed the ONS’s transparency and advanced notice about the errors.
ES.15 A range of other well-established survey and administrative data sources are used to produce the PPIs. These sources have been used for a long time, but their suitability and quality have not been fully reviewed in recent years (Requirement 5).
ES.16 The business prices team engages proactively with users of the PPIs within and outside ONS, for example, through regular bilateral meetings and cross-government business prices groups. Users who use the statistics for the purposes of forecasting and indexing contracts told us they are generally satisfied with quality and timeliness of the data.
ES.17 Currently, ONS publishes a large number of indices, some of which are likely to be poor quality due to the small sample sizes, and some are likely to be unused. The business prices team should rationalise the number of indices it produces, focusing on producing high quality indices that meet user needs (Requirement 7).
List of Requirements
Requirement 1:
To improve the quality of the PPIs and ensure that they meet users’ needs, ONS should undertake a review of the necessary sample size and sample optimisation for the PPI, EPI and IPI surveys by July 2024, and update the samples accordingly by July 2025. In the meantime, ONS should consider what remedial changes it can make to improve sampling arrangements sooner than 2025.
Requirement 2:
To safeguard the quality of PPIs, ONS should publicly commit to clear and achievable transformation plans for developing a robust, flexible and sustainable producer price inflation statistics system. This should enable RAP principles to be applied throughout and allow new sources to be used and new methods to be implemented. ONS should publish and promote the plans as part of the wider PPI improvement plans by September 2023.
Requirement 3:
Within six months of moving to the Statistical Processing Platform, ONS should review the PPI data validation processes and checks to ensure they provide an appropriate level of quality assurance and are adaptable to the prevailing general level of price increases.
Requirement 4:
To improve its understanding of revisions and minimise their impact on quality, ONS should carry out a revisions analysis every year. Where revisions are found to be significantly different from zero, ONS should investigate their source and, where necessary, make appropriate improvements to the methods for producing PPIs.
Requirement 5:
To ensure the continued suitability of data sources used to produce PPIs, by July 2024, ONS should review the suitability and quality of all current data sources and improve its understanding of the quality assurance carried out by data suppliers. To help users understand how PPIs are compiled, ONS should add a high-level process map to quality documentation explaining how the different sources contribute to the final estimates.
Requirement 6:
To ensure the best available methods are being used, by July 2024, ONS should review its imputation methods, including assessing whether they are still fit for purpose and not introducing bias.
Requirement 7:
To maximise the usefulness and quality of the published indices, and optimise the use of available resources, ONS should rationalise the number of indices produced by July 2024. It should take into account users’ needs and sample size limitations.
Requirement 8:
To enhance transparency and provide reassurance to users about quality, by July 2024, ONS should ensure that its published information about data sources, methodology and quality assurance covers all aspects of the production of the statistics and is suitable for a range of users. ONS should review and update this information whenever needed to reflect current processes.