Ed Humpherson to Jonathan Athow: Review of population estimates and projections produced by ONS

Dear Jonathan

Review of population estimates and projections produced by ONS

As you are aware, we recently undertook a review of population estimates and projections produced by the Office for National Statistics (ONS), against the Code of Practice for Statistics. This letter summarises our findings and recommendations.

Background

Our review was initiated in response to concerns raised with us in November 2020 regarding the population projections and mid-year population estimates for Coventry. The concerns were around the perceived inaccuracies of the population estimates on which the household projections and subsequent housing need are based. It is not within our remit to regulate operational decisions made by government or local authorities, nor to form a judgement on decisions about government policy. The population estimates and projections are important data with implications for many other statistics and influence decision making. The projections feed into local planning decisions which can have a long-term commitment and therefore the impact of issues can have far reaching consequences.

Coventry

Our review considered the population estimates and projections against the Code, independent of the specific issues raised with us concerning Coventry. In carrying out our research and speaking to a number of demographers, academics and representatives from local government, we found that the population estimates did appear to be inconsistent with, and potentially higher than local evidence suggests in some smaller cities that had a large student population. While ONS acknowledged that there are limitations with data on highly mobile groups such as students, ONS did not adequately consider Coventry’s concerns and more needs to be done to investigate the root and scale of the issue associated with students and outward migration. You should work with areas such as Coventry and independent demographers as you develop new population estimates through the transformation programme.

Summary of our findings

We have grouped our findings into themes focusing on methods, uncertainty and uses of the data, which are set out in more detail in our report.

ONS collaborates with a range of experts to determine the methods, data and assumptions which underpin the population estimates and projections. At the national level, the approach taken by ONS is fit for purpose and is highly regarded internationally. At the subnational level, the accuracy of estimates is variable due to factors such as the size and mobility of the population in a given area.

One area of challenge has been migration, where there are limitations in the data available. You have sought to address this challenge by introducing some methodological fixes, such as the way students leaving university are identified. However, more needs to be done to investigate the scale of the issue as it relates to the current population estimates and projections particularly for cities with large student populations. More broadly, as ONS is seeking to make improvements to its population migration estimates, you must understand and address known issues such as the nature of international student emigration and the impact of that on local population profiles. It will be important for you to draw on the knowledge and experience of ONS teams working directly on the current population estimates and projections in understanding these issues. We are pleased to see you recently published several updates about the progress of this work, including the development of admin-based estimates and the statistical modelling approach.

ONS has a number of methods for quality assuring the statistics, including deep dives, triangulation of data it holds and comparisons against historic data. You have developed a range of variant projections to cater for the different uses of the data. We suggest you develop case studies of how these variants are being used in practice to promote their use more widely, as some users were unclear on which variants would best cater to their needs.

ONS engages regularly with experts, academia and other users, and participates in relevant events and conferences. We found that while ONS is good at sharing its work outwardly, there is room for improvement in the way it takes on board feedback and handles challenge. We would like to see ONS’s population estimates and projections team be more open and responsive to issues when they first arise and view challenge as an opportunity to improve outputs and not a criticism of its approach. We recognise that the team are balancing competing priorities, but a more open and constructive approach to responding to user feedback would create opportunities for ONS to continually improve the population estimates and projections, and ensure users feel listened to.

Summary of our recommendations

We have identified a number of actions that should be taken by ONS in response to our findings. These are provided in more detail in the report and are summarised below.

Improving methods
• ONS needs to investigate the root and scale of the issue associated with cities with large student populations and communicate its findings publicly, to support the appropriate use of the existing data.
• ONS needs to integrate a more flexible and responsive approach to methodological changes in its design for admin-based population estimates, working with its external partners, so that improvements are more timely.
• ONS should collaborate with others to incorporate local insight and carry out sensitivity analysis to enhance its approach to quality assurance.
Enhancing communication
• ONS should be open with users about its short-term solution to bridge the gap of migration data until the administrative data alternative is fit for purpose and ready to use.
• ONS should provide more specific guidance on interpreting the levels of uncertainty associated with the population estimates and projections, to help support the appropriate use of the statistics.
• ONS should develop case studies of where the variant projections have been used in practice and beneficial to users, to support their use more widely.
Embracing challenge
• ONS should take a more open and constructive approach to responding to user feedback, by improving its complaints procedure and viewing challenge as an opportunity to improve the statistics and outputs.
• ONS should collaborate with experts to frame the statistics for different audiences and scenarios.

Next steps

We expect ONS to action our recommendations and also reflect on our findings when developing its new approach to population estimates and projections. The Census will help determine the scale of the apparent overestimation, but you should ensure the issues we have highlighted are addressed in the design of the new admin-based approach.

To support the delivery of the recommendations, you should focus on determining whether the issues raised here have an impact on other official statistics. It would be worthwhile considering whether colleagues in the ONS Centre for Subnational Analysis can carry out some further analysis to support you in meeting the recommendations. We would also encourage you and the team to engage with the devolved administrations, through your existing working level partnerships, to assess how the issues concerning students and emigration may impact their estimates and projections.

During the course of this review, we received further concerns from other local areas regarding the way population estimates feed into the household projections from which housing need is determined. Our findings will support our imminent compliance checks of household projections in Scotland and Wales, and will inform the direction of further regulatory work we carry out on household projections in England.

We will continue to monitor the development of these statistics and support you in working towards our recommendations. We ask you to report back to us with your plans for addressing these recommendations in July 2021.

Yours sincerely

Ed Humpherson
Director General for Regulation