Dear Iain,

As you are aware, we recently brought forward our planned compliance check of Overseas Travel and Tourism statistics following a letter from Tourism Ministers for the UK, Welsh and Scottish Governments. The letter raised concerns about estimates of inbound visitors from the International Passenger Survey (IPS), most notably the reduced sample size which has resulted in widening confidence intervals and the fact that the Ministers did not feel ONS had given their concerns sufficient priority.

Overseas Travel and Tourism statistics play an important role in understanding the tourism industry in the UK. These statistics are highly valued by users, who often rely on them as a unique source of information. Our review focused on two issues relevant to the Quality and Value pillars of the Code of Practice for Statistics: the IPS sample size and how the IPS team engages with users. Users expressed a keen interest in seeing improvements in the quality and communication of the statistics. We appreciate the helpful and open way that your team engaged with us during this process.

We have presented our detailed findings and requirements in the annex to this letter. In summary, they are as follows.

• ONS does not have a full understanding of who uses the statistics published in the Overseas Travel and Tourism release, or how they are used. This means the IPS team is unable to judge the quality required for a range of users and how information should be presented to meet their needs. Overseas Travel and Tourism statistics provide a valuable contribution to the UK National Accounts, which require data at a UK-level. The team have a good understanding of these needs and have prioritised them when making decisions about the survey. However, the prioritisation of UK-level estimates in terms of quality and presentation does not reflect the requirement of the majority of other users who need data for smaller geographic areas. ONS should develop a more complete understanding of how Overseas Travel and Tourism statistics are used and by whom, so that it is able to balance different user needs. Where not all user needs can be met ONS should be clear on the priorities and rationale for its decisions.
• ONS should take a more proactive approach to engagement and communication with users outside the IPS Steering Group. The IPS Steering Group functions as a forum for sharing updates from ONS but is not as effective as it should be in gaining input on proposals or eliciting feedback from users. Furthermore, in line with Finding 2 from our compliance check on the IPS in 2018, we found that users outside the Steering Group are still not informed about planned developments and changes to methods and outputs. We are pleased that your team is now considering how it can engage with users in a more active way. We encourage ONS to engage with a broad range of users in order to maximise the public value of these statistics. We suggest that the IPS team continue to work with the GSS Good Practice Team to develop their understanding of effective user engagement and to implement these approaches.
• Users are concerned about the uncertainty of estimates. For example, in the most recent quarterly data, the confidence intervals for numbers of overseas visitors to Scotland and Wales are +/- 24.2% and 29.6% respectively. Similarly, the confidence intervals for expenditure by overseas visitors to Scotland and Wales are +/- 26.7% and 43.8% respectively. While annual data have narrower confidence intervals, in general, these confidence intervals have increased from previous years due to the decline in sample size and have impacted the ability of users to draw reliable conclusions. We recognise that there may be limits on the potential to improve the IPS to the extent many users want and that substantial changes may depend on additional investment. We encourage ONS to be clear about any limitations with their users. We note that response rates, but not sample sizes, are published by ONS.
• We welcome the recent implementation of a Project Board and formal workstreams to investigate issues relating to sample size which have impacted uncertainty. We acknowledge that there are several complex factors which have contributed to the reduction in sample size and which require substantial investigation alongside other competing priorities. We understand that a key priority for the team is the completion of the critical work to correct the imbalance within the IPS which directly impacts the quality of Travel and Tourism Statistics. However, ONS has still been slow to respond to emerging issues and has not explained to users what has happened to sample sizes in a clear and timely way. ONS should complete investigations into the sample decline and publish the results of this within the next nine months.
• We welcome the recent work by the IPS team to improve the provision of management information and encourage ONS to ensure that adequate and appropriate resources are available for monitoring sample sizes and responding to emerging issues. We consider that better ongoing monitoring of management information in the past would have enabled issues, such as those regarding sample size, to have been identified and managed sooner.

We consider that Overseas Travel and Tourism statistics should retain National Statistics designation, provided the following improvements are made to ensure compliance with the Code of Practice.

By January 2020

• ONS should report back to us on its plans to improve user engagement and provide an update on its investigations into the sample size drop.

By April 2020

• ONS should have implemented improvements in user engagement.

By July 2020

• ONS should publish better information regarding quality and methods.
• ONS should report back to us on how it is ensuring that adequate resources are available for monitoring sample sizes and responding to emerging issues.

We will continue to work with your team to provide detailed feedback and suggestions from our investigation as they work to implement these improvements. On the basis of progress made by July 2020 we will decide whether a full assessment of these statistics is necessary.

Whether or not statistics are fit for purpose depends on how they are used. Therefore, the National Statistics designation is awarded to the Overseas Travel and Tourism output rather than the underlying data source, the IPS itself. We will continue to assess and communicate with you about tourism and migration statistics derived from the IPS separately. However, we note ongoing quality issues which affect both and encourage you to continue to build greater collaboration between your teams to ensure the data are well understood for the range of purposes they are used for.

We are mindful of the challenges around official statistics derived from survey data more generally and therefore are considering a systemic review on the value of surveys as part of our future work programme. In the meantime, I encourage you to review the resources allocated to the IPS as you address the findings from this review.

Yours sincerely

Ed Humpherson
Director General for Regulation

 

For annex, see PDF.

 

Related Links:

Sir David Norgrove to Tourism Ministers (October 2019)

Letter from Sir David Norgrove to Tourism Ministers (August 2019)

Letter from Tourism Ministers to Sir David Norgrove (August 2019)