What we Found

Quality

ONS takes a sensible approach to measuring the population

The methodology documents published by ONS on the population estimates and projections, at both the national and subnational levels, are very detailed and informative. They include information on how the estimates and projections are derived and record any changes that have been made to the methodologies. The impact of these and details of data sources used for quality assurance purposes are also documented.

At the national level, we consider the approach taken by ONS to produce population estimates and projections is fit for purpose. The choice of methods, data and assumptions has been supported by expert advice from demography and academic partners. The methods are viewed by demographers and statisticians as strong internationally and ONS is seen as being at the forefront of addressing the complex challenge of measuring the population, in the absence of a national identification register.

At the subnational level, it is widely understood by users that the accuracy of the estimates will be variable due to factors such as the size and mobility of the population in a given area. We found that in some smaller cities that had a large student population, the population estimates did appear to be inconsistent with, and potentially higher than local evidence suggests. ONS’s population estimates team recognises that areas with high population churn are harder to estimate and it has introduced a number of methodological changes, which are detailed later in this report, aimed at mitigating this issue. However, these fixes do not appear to have fully addressed the perceived overestimation of these groups in some areas.

The mid-year population estimates (MYE) are produced annually and the population projections once every two years. Following each decennial Census, the estimates are rebased to be in line with the Census population estimates so at this point they are at their most reliable. Each year thereafter, the cohort component method is applied to roll forward the estimates, taking account of the three base components of births, deaths and migration. Whilst the Census provides the most complete data on the population, the timeliness of the data affects the quality of estimates in the interim years and there are known coverage issues for some groups such as young men and those in houses of multiple occupancy.

ONS works with expert partners to review and update the assumptions which underpin the methods used to produce the population estimates and projections. Where assumptions are made based on historic trends which do not reflect current behaviour, there is a risk that ONS builds in systematic bias by carrying through an error into the rolled forward estimates and then subsequently the projections, which compounds the effect of the error. For example, where the female student population in an area is over-estimated, as this group is of child-bearing age, the rolled forward estimates will impact the fertility rate which further exacerbates the issue.

ONS has introduced a number of methodological fixes but the impact of these changes is still unclear

ONS’s population estimates and projections team has tried to address some of the methodological challenges it faces in relation to the migration component that feeds into the population estimates, despite it being no easy feat. For example, it introduced the Higher Education Leavers Methodology (HELM) to improve estimating the internal movement of students on leaving university and a change in the modelling approach for estimating international outward migration. When the results from the 2021 Census are available, ONS can assess the impact of the steps it has taken.

Whenever ONS has made these changes, it has carried out a range of user engagement activities to test the approach with users. For example, it has previously run touring roadshows in an effort to talk directly to users and experts. Some users told us that previous fixes that have been made to the methodology have had unintended consequences on other areas. For example, a fix introduced for international migrants arriving in London who were previously being recorded in Westminster rather than the borough they intended to stay in, appears to have led to errors in the way international migrants are recorded elsewhere.

ONS told us it does consider whether there are systematic issues when it receives complaints but that it is hard to see if the impact is likely to be temporary until a few years down in the line. Thought it does not have a specific rule regarding the number of changes it makes, ONS told us that it aims to limit methodological changes to once per decade to prevent disrupting the time series. Whilst we understand that it would not be sensible to have too frequent changes to the methods underpinning the estimates and projections, ONS has a responsibility to prevent systematic bias being built into the statistics and should have a more flexible approach to addressing issues where the impact is felt across a number of areas and over time.

The pandemic has sharpened the focus on the definitions of migration and population in these statistics, and what the population estimates and projections were designed to do. The definitions are premised on stability which has raised questions about whether they are fit for purpose in this period.

The pandemic and the UK’s departure from the EU have both caused shocks to migration patterns in the UK. It is difficult to unpick how these shocks interplay in the data and to what extent they have individually impacted migration behaviour. The standard cohort component methodology is designed around stability but doesn’t deal well with shocks to the system. ONS has been exploring this issue and is looking to communicate its work in this area with users to draw out the insights from how our understanding of population has changed during the pandemic.

Migration data continues to be a challenge for ONS

We found that users generally had no issues with the source data used for births and deaths which feeds into the population estimates. However, there were some strong concerns expressed about the potential bias in the emigration data for some groups, such as international students, that are hard to count.

The accuracy in the internal migration (i.e. movement within England) component of the estimates can be problematic as it is largely dependent on General Practice (GP) registration data which is known to suffer from data quality issues. For students and young professionals in particular, they may not re-register with a new GP when moving to a new area until they need to use its services, or they may not re-register at all. Some individuals may also choose to register with a GP close to their work rather than their home.

The international migration component that feeds into the population estimates has been predominantly based on ONS data derived from the International Passenger Survey (IPS) with additional input from administrative sources. ONS has acknowledged the limitations and weaknesses of using IPS data for international migration and continues to work to develop new and exploratory methods and data solutions to improve these statistics. This work has been expedited as the IPS was suspended in March 2020 as a result of the pandemic and no long-term migration estimates have been produced since the last publication covering long-term migration in the year ending March 2020.

International outward migration has historically been hard to estimate as there are few and only partial data sources which do not provide a complete picture. ONS’s population estimates and projections team takes a modelling approach to estimating emigration but the outflow of people is more uncertain. This creates issues for capturing international students who return home after their studies.

ONS is taking a joined-up approach to tackling the challenges in measuring migration, population and the labour market during the COVID-19 pandemic. It outlined its approach to overcoming these challenges in a blog to inform users about its plans. As part of its longer-term transformation programme, ONS is looking to make greater use of administrative data to enhance these statistics but there is no quick solution to addressing this data gap. The pandemic has also created a greater time lag for some of the administrative datasets that it was intending to use for measuring migration, which were already lagged due to the nature of measuring long-term migration, and it is now exploring greater use of modelled estimates and nowcasting for migration data.

We are pleased to see that ONS has recently published several updates about the progress of this work, including the development of admin-based estimates and its statistical modelling approach. It is positive to see ONS share its thinking and it should continue to communicate its progress in overcoming the challenges with migration data, particularly around international outward migration.

ONS could think more creatively about its approach to quality assurance

ONS has processes in place to quality assure the data and methods used to produce the population estimates and projections. This often involves ONS’s population estimates and projections team triangulating data it holds and making comparisons against previous trends. ONS receives advice from a panel of experts in the fields of fertility, mortality and migration, which helps it determine the underlying assumptions. ONS also publishes an interactive mapping tool to allow users to compare subnational population projections (SNPP) with other areas and projections.

The team in ONS carries out ‘deep dives’ into areas which have counter intuitive results and will use external sources to investigate the issue. For example, it sometimes uses Google Maps to look at changing street pictures or data from the Higher Education Statistics Agency (HESA) to look at changes in student numbers in a given area.

Although ONS does look to triangulate data sources to quality assure the estimates or to investigate issues, we found that it could be more open to local sources of information where it overwhelmingly disputes the population estimates. We acknowledge that it is not practical for ONS to do this for all areas and that one source of information will not provide a complete picture for an area. However, where substantial local evidence points to a trend contrary to the population estimates, this should be investigated as a priority during the quality assurance process.

The systems which ONS is working with enable it to carry out sensitivity analysis. We found that the Greater London Authority (GLA), who conduct their own analysis of population estimates and projections, make good use of sensitivity analysis to understand the impact of different assumptions and scenarios and publish the results. We would encourage ONS to enhance its approach to quality assurance by carrying out and publishing relevant sensitivity analysis.

ONS has taken steps to communicate uncertainty

ONS’s population estimates and projections team has made a concerted effort to communicate the statistical uncertainty of the population estimates and projections, including presenting confidence intervals and caveats. It also publishes a range of variant projections, which are explained in more detail in the next section of this report, to provide an indication of the ‘fan of uncertainty’ around its assumption setting.

Despite this, the language used to describe the statistics, for example ‘the number of women has increased by’ rather than ‘is estimated to have increased by’, and lack of rounding in the figures implies a precision that doesn’t exist and can therefore be interpreted as an exact figure rather than a central estimate.

We found that ONS could do more to interpret the uncertainty for non-analytical users to highlight the robustness of the data for practical uses. For the projections in particular, there should be clearer guidance on the uncertainty or ‘shelf-life’ of different length trajectories so that decision makers can determine the appropriate projections to use to inform longer term strategies.

The effects of the pandemic and the UK’s departure from the European Union are challenging for population statistics. ONS’s population estimates and projections team is currently collaborating with international colleagues to share insights and explore the best way forward in dealing with these challenges.

ONS produces a range of variant projections to meet the range of user needs for population projections

ONS produces a range of variant population projections in addition to the principial projections. These variant projections are based on different assumptions of future fertility, mortality and migration which users find helpful as it allows them to select the projection which most suits their needs for the context which they are working in. These variants also provide projections which are based on different lengths of historic data so that users can benefit from the trend length which suits their purposes for the projections.

ONS’s recent publication on Early Indicators of UK Population gave more prominence to the effects of applying different migration variant projections in light of the COVID-19 pandemic. This was a useful way of presenting and communicating the different scenarios to users.

For the internal (within-England) migration component needed for the SNPPs, the variants are also based on the number of years used for the base period. Previously, ONS used the latest five years of records as the basis for its principal projection but changed in the most recent projections to the latest two years of data and also released a 5-year and a 10-year-based alternative. To illustrate, a projection that has been produced using 5 years of past trend data will be less suitable for making planning decisions for the forthcoming 15 to 25 years than one that has 10 years or even 25 years of historical trend. The more years of past trend data that are included, the more stable the projection for future local planning needs. However, there will be other situations where a projection based on 5 years of past data will be suitable.

The pros and cons of switching from a 5-year to 2-year base for the principal projections are likely to vary depending on the use to which they are put. ONS suggested that the latest 2 years might better represent the future as the latest methodological changes are included but suggest that users should refer to the variants if this was felt not to be the case.

ONS’s population estimates and projections team told us that its approach to producing the variant projections is customer led and the team offers advice on how to use them. While users we spoke to told us that they find the sub-national variant projections useful, we found that those involved in local planning decisions lack the confidence to use the variant projections as they are not seen as carrying the same weight as the principal projection. At the Local Authority level, the process for using the SNPPs is built around using the principal projection. There is also a reluctance to use the variant projections where there are known issues in the underlying population estimates, for example the overestimation of students, as this can lead to the variant projections presenting implausible scenarios.

Some users also told us that it would be beneficial to have projections which are based on more than 10 years’ worth of data, as some government departments deliver projects with up to 25-year timescales that would benefit from a longer trajectory – for example transport planning.

ONS has recently announced its plans for 2020-based interim national population projections (NPPs) following a consultation to assess user need. The UK Census Committee (UKCC) decided that, in order to meet user needs identified through this consultation and to support the forthcoming State Pension Age Review, a principal national population projection only will be published for each UK constituent country and for the UK as a whole, with no variant projections.

The feedback from users particularly on variants will be valuable beyond decision making for the 2020-based NPPs and ONS should consider how it feeds these through to its plans for future developments.

Value

Users feel ONS’s data cannot be challenged and ONS could be more open in its approach to responding to user feedback

ONS’s population estimates and projections team engages regularly with experts in demography and subject matter experts for the components which underpin the population estimates. The team attends biannual Central and Local Information Partnership (CLIP) meetings with local authority users, and also engages with the UK Population Theme Advisory Board. The team remains alert to emerging interests through engagement with social media, newsletters and through participation at relevant population statistics events and conferences such as those run by the British Society of Population Studies.

ONS has convened several user groups for migration statistics, including an expert group made up of key technical experts in the migration field, and a Government Statistical Service steering group made up of senior representatives from relevant government departments. These groups provide ONS an opportunity to test its research, provide challenge and steer developments for migration statistics.

Where issues have been raised about the statistics, ONS has offered meetings with these users, to listen to their views and explain how the statistics are produced. It told us that it engages fully with all correspondence relating to complainants until the issues are resolved.

However, there seems to be a disconnect in how much ONS feels they have supported users and how well users feel listened to. We found that the way ONS engages can at times be perceived as ONS being selective in its choice of points to respond to and that the engagement can become closed if ONS feels it has already addressed the concerns elsewhere.

Users do not feel there is a reasonable process to challenge the estimates even when presenting local administrative data to illustrate their points. While we acknowledge the competing priorities that ONS must balance, inviting and responding to external scrutiny is an important way for ONS to improve its work. ONS could do more to involve local decision makers in the production of the statistics so that they can aid understanding and provide insight which may be useful for enhancing the methodology.

ONS needs to be a vocal advocate of using these statistics appropriately to serve the public good

The population estimates are vital and have a very widespread use in non-Census years. They are used for weighting or as the denominator in the production of many other statistics, and they feed into the population projections that are in turn used for many aspects of local planning.

There is a potential risk to other statistics that use the population estimates if a systematic bias in the estimates (even if one segment of the population) is relevant for a particular topic area. A second issue comes from the nature of the method for disaggregating the national level to local areas, when the disaggregated data are constrained to match the national level, inherent biases in the data could lead to skewed local area data.

Where the estimates for some Local Authorities might not reflect the local situation well enough, it can have a knock-on effect well into the future. This issue is compounded by the fact that most planning policies are designed around having one figure to reflect need and do not take into account the uncertainty of that figure. We heard from users that there is a lack of analytical resource within most Local Authorities to question the figures and therefore the principal estimates and projections are interpreted as precise and not open to challenge. This can lead to local planning interventions being mismatched with local needs.

The population projections inform the household projections. The Ministry of Housing, Communities and Local Government (MHCLG) made a policy decision to specify that Local Authorities use 2014 household projections as part of the standard method for calculating house need, rather than the more recent 2018 household projections produced by ONS. This means any methodological changes made by ONS to improve the population estimates since 2014 are not reflected in the statistics which inform housing need. For some Local Authorities, this means the over-estimation of population in certain age groups is driving policy targets in a different direction to local priorities.

ONS produces statistics with integrity and impartiality, in line with the Code of Practice for Statistics. It is not the role of ONS to regulate how the statistics are used to inform policies, but it is its role to advocate for the appropriate use of the data. ONS must take responsibility for ensuring the strengths and limitations of the statistics can be appropriately understood by those who intend to use them, particularly where the use of the statistics may have significant long-term impacts on those affected by the policy. We recognise that ultimately ONS cannot control the decisions of policy makers but ONS should be vocal in speaking up against those who choose not to use the most up to date and comprehensive figures, where there is not a reasonable argument for them to do so.

Trustworthiness

ONS is seen as a capable and informed statistics producer

ONS is seen as a credible and reliable statistics producer, whose methods are robust and highly regarded internationally. At the local authority level, this means the estimates are sometimes seen as “fact” rather than estimates, and the level of uncertainty associated with them is not sufficiently considered. This relates to our findings around the wider lack of understanding of how to interpret uncertainty.

ONS could be more transparent about its approach in dealing with challenge around the population estimates and projections, as we have set out earlier in this report.

We do not have any recommendations concerning the Trustworthiness pillar of the Code.