Dear Iain,
Thank you for your letter of 21 August setting out your recent progress and findings from your development work on migration statistics. We appreciate your openness and transparency about the limitations of long-term international migration estimates and the efforts you and your team are making to improve them. The population and migration statistics transformation programme is important work and we are pleased to see the pace you have moved at since identifying concerns in your last published update in June.
We have concluded that the Migration Statistics Quarterly Report (MSQR) should no longer be designated National Statistics, including long-term international migration estimates. This letter sets out our expectations for your proposal to publish the MSQR as experimental statistics tomorrow.
You have set out a compelling case for the greater insight provided by drawing on multiple sources of data in your long-term international migration estimates, but you have also highlighted significant remaining uncertainty, particularly for post-2016 estimates. It is critical that decision makers, including politicians, businesses and the public, have robust and reliable migration estimates. It is for producers of statistics rather than the regulator to determine whether or not to classify an output as experimental statistics. However, based on what you have told us about your planned developments, your findings so far, and the need to engage users heavily during this development phase we consider it appropriate to classify long-term international migration estimates in the Migration Statistics Quarterly Report (MSQR) as experimental statistics.
We urge ONS and other government departments involved in the transformation programme to hasten progress and prioritise this development work to improve estimates, whether through data sharing, collaborative research work, or other sharing of expertise and resource. If there are any aspects where we can support your endeavours, then we would be happy to work with you to achieve this.
On 31 July, we issued a statement regarding the quality and value of international migration estimates. In that statement we called for urgency in completing ONS research on comparative data sources, transparency on the outcomes of this research and clear signposting of quality issues. We have set out our view on ONS progress in each of these areas in advance of tomorrow’s publication.
“ONS must complete its research into the differences between the migration estimates derived from the IPS and APS without delay”
We welcome the pace at which ONS has moved to develop migration estimates since your June update, including drawing on additional sources to support your conclusions and better understand limitations with the IPS and APS.
“ONS must be transparent and open regarding the outcomes of this research, and continue to inform users about the extent of any quality issues as it becomes aware of them”
We note the publication of the research undertaken to date to investigate the coherence of different sources of migration data and the resultant adjustments to long-term international migration estimates you plan to introduce. Measuring migration is extremely complex and the use of additional sources of information should improve the accuracy of your estimates of migration, but you also highlight considerable remaining uncertainty.
As you continue to explore how other data sources may better improve international migration statistics, we expect you to keep users of the statistics up to date on your research and findings.
“ONS must continue to clearly signpost quality issues within future statistical publications, providing guidance for users on appropriate interpretation”
Ceasing to publish as National Statistics and reclassifying the release as experimental statistics provides a clear statement of the development status of the estimates. It helps you to inform users about limitations and quality concerns while seeking their input. The testing of experimental statistics allows producers to gain a good understanding of the quality of the statistics, including their accuracy and reliability. Users are central to this process and we expect you to continue to engage with a range of users of migration statistics.
Alongside the reclassification of these statistics, users must be supported and provided with guidance on the appropriate interpretation of the new estimates, at aggregate and disaggregate level. I note from your letter the uncertainty around EU8 immigration estimates since 2016, given the data currently available to you. ONS should include prominent guidance on the use of post-2016 estimates and the likelihood of further revisions in future statistical releases.
In the context of your development work we consider experimental statistics to be the most appropriate classification for your long-term international migration estimates. The Code of Practice applies to all official statistics and encourages innovation and improvement and highlights the need for statistics to remain relevant. Experimental statistics are official statistics in development and our recently published guidance on experimental statistics encourages producers to keep alert to the developments and opportunities in evolving technologies and methods, to improve estimates and better meet the public good. Annex A sets out further detail on why we consider the experimental statistics classification appropriate for your long-term international migration estimates and provides more information about our remaining concerns.
Once the MSQR is published tomorrow, we will review the release to see how you have implemented changes to methods to incorporate new data sources and how this has been explained to users. It should be clear to users that these are not National Statistics. For the experimental status to apply to these estimates it should also be clear to users of the statistics that estimates produced using the new methods are the headline statistics and users should be adequately supported in their use and interpretation of these figures. For other statistics included in the MSQR, such as Home Office data on long-term visas, you should be clear on the separate designation of these statistics.
I would also like to take this opportunity to make you aware of our review of National Statistics designation, which we have just initiated. As part of this review we will be looking at what National Statistics status means to users. It will also consider how the designation should apply in an increasingly complex world in which producers of statistics are undertaking continuous innovation and drawing on a wider range of data sources. My team will engage with you as we develop this work.
I would like to acknowledge the efforts of your team, particularly their openness and willingness to keep OSR up to date on development work over recent months. We look forward to further updates as you work to deliver your transformation and expect these to reflect the need for urgency and collaborative working to deliver high quality migration statistics that serve the public good. We will welcome an assessment request when you consider your current work programme is complete, which we expect to be in summer 2020.
I am copying this letter to Sir David Norgrove, Chair of the UK Statistics Authority.
Your sincerely
Ed Humpherson
Director General for Regulation
Annex A: Our Reasoning
Why is experimental statistics an appropriate classification for long-term international migration estimates and why is this change being made now?
ONS, alongside other government departments, has been working to develop better long-term international migration (LTIM) estimates as part of its population and migration transformation programme.
LTIM estimates have historically been based on data from the International Passenger Survey (IPS), with some adjustments made for specific issues such as asylum seekers, resettlement schemes, changes of intention and flows to and from Northern Ireland. Work by OSR and ONS has highlighted limitations of the IPS, an intention-based survey, with the current demands on migration statistics pushing the IPS beyond what it was originally designed for – a survey of travel and tourism.
In June 2019 ONS published a progress report which investigated the divergent patterns in what the IPS and the Annual Population Survey (APS) tell us about migration. This led to an accelerated programme of work to try and better understand these differences, including drawing on other sources of administrative data. The details of this work are outlined in the ONS research report published today.
ONS concluded that to get the best overview of trends in migration it needs to look at all available data sources, and it is no longer adequate to base estimates so predominantly on the IPS without consideration of other sources. ONS has therefore decided to introduce preliminary adjustments to its previously published headline measure of migration. This includes:
- An uplift to non-EU emigration (outflow). Analysis undertaken by ONS suggests that because of uncertain intentions in this group the IPS has underestimated the number of non-EU students leaving the UK at the end of their studies. This results in a reduction in non-EU net migration.
- An uplift to EU immigration (inflow). ONS analysis suggests that it has previously underestimated immigration for the EU8[1]. An adjustment has been made up to 2016. This increases net migration from the EU.
The adjustments are informed by data from the Home Office, Department for Work and Pensions and Higher Education Statistics Agency. These data have been used in previous research aimed at gaining a better understanding of migration.
The new information has suggested there are meaningful improvements to be made by drawing on a range of data sources. It would therefore no longer be appropriate to continue to publish LTIM estimates as National Statistics using the historic approach. The inclusion of new data sources to inform estimates of long-term international migration marks a significant change in the approach to producing these estimates. It is an important step in the transformation journey, but far from the end. ONS expects to get access to new data sources in future, which should help it further refine the estimates.
We therefore consider LTIM estimates produced using this new approach as statistics which are in development, requiring continued investigation and evaluation by ONS. For this reason, we agree that it is most appropriate for MSQR to be classified as experimental statistics. The classification highlights the development of the statistics as part of the migration statistics transformation programme. It also highlights the potential for further adjustments and a wider degree of uncertainty as the methods and processes are developed, established and verified.
Are these estimates the best available?
Quality issues with LTIM estimates based on data from the IPS are well known and acknowledged by ONS, these include:
- Limitations resulting from small numbers of long-term migrants in the survey sample. Many variables cannot be disaggregated far below headline estimates before being subject to unacceptable margins of error – limiting their use for local level analysis.
- A declining migrant contact rate as a proportion of the achieved IPS sample, particularly for outflows/emigrants – related to steadily increasing flows of overall travellers through UK ports. This makes it less likely long-term migrants will be sampled and leads to increased sampling error around migration estimates.
- The disconnect between people’s stated migration intentions to IPS interviewers, and their observed behaviour.
- Other inconsistencies between IPS measures of migration flow, and estimates of migration stock, captured by other sources such as the APS.
While considerable work has been carried out to understand how to address issues with the IPS, a lot of uncertainty remains. The preliminary adjustments to LTIM estimates should offer users a better estimate of migration than the previously predominately single-survey source. The multi-source approach should support ONS in addressing some of the limitations of the IPS.
The combination of the adjustments made means that the headline estimates for net migration are broadly unchanged, but there is a larger impact on breakdowns. For example, in the year ending March 2016 – which includes adjusted data for both inflow and outflow – net migration is around 1 per cent higher than the previously published estimate. For the EU and non-EU adjustments, the estimates show a similar trend over time, but there are larger differences between the original and adjusted series. For example:
- EU: In the year ending March 2016 – the latest point at which ONS have adjusted data available – EU net migration is around 16 per cent higher than the previously published estimate.
- Non-EU: In the year ending March 2016, net migration is around 13 per cent lower than the previously published estimate. In the year ending December 2018, it is around 8 per cent lower.
The weakest area of understanding is for EU migrants post-2016. There is insufficient evidence to make a firm judgement. ONS therefore publishes a best assessment of EU net migration since 2016 as still adding to the population, while acknowledging other sources suggest that the IPS may be a slight overestimate.
Measuring migration is complex and it appears that the adjustments outlined by ONS lead to a better estimate of migration than was previously available. The consistency at the headline level provides some reassurance in the historic series. However, further work is urgently needed to reassure users that they can have full confidence in the existing estimates, particularly post-2016.
The use of more administrative data in future should support improvement and mean more insights into migration patterns may be gleaned. Migration estimates in future statistical releases are likely to be subject to further revision as ONS’s work progresses.
What we expect from ONS
Given the importance of long-term international migration estimates and the remaining uncertainty surrounding them we expect ONS to hasten progress and prioritise this development work to improve estimates, whether through data sharing, collaborative research work, or other sharing of expertise and resource.
While being published as experimental statistics ONS should:
- be clear about how it will involve users and method or topic experts in the development and testing of the statistics
- help users understand the scope and nature of the development, and why it is needed and important
- be clear about how it will decide whether the statistics are of sufficient quality and how it will determine that the statistics meet users’ needs
- set out the timeframe for the development, giving a clear indication of expected milestones
- liaise with OSR on reassessment of the statistics
Users of these statistics must be supported and provided with guidance on the limitations and appropriate interpretation of the new estimates.
[1] EU8: Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia
Related Links:
Iain Bell to Ed Humpherson (August 2019)