Ed Humpherson to Ian Knowles: Rapid Review of DfT Transport Statistics during the COVID-19 pandemic

Dear Ian

COVID-19 TRANSPORT DATA

I am writing to endorse the approach taken by the Department for Transport in producing statistics on the change in transport use during the COVID-19 outbreak. I would like to congratulate everyone involved for their hard work to produce these timely and valuable statistics under extremely challenging circumstances.

These statistics are currently published at the discretion of the Cabinet Office for the government’s daily media briefings. Although the full data series is ultimately made available, it is released on some days and not others. Given their public value, the Department for Transport should commit publicly to a pre-announced release schedule. Doing so should give ample time for the Cabinet Office to decide what and how to present in the daily media briefings and would demonstrate trustworthiness.

My team has conducted a rapid review of these statistics. We have reviewed the extent to which these statistics have been produced in accordance with the Code of Practice’s Trustworthiness, Quality and Value pillars, while taking account of the pressures on you and your teams to deliver near-to real-time statistics about a rapidly changing situation. A summary of our findings and recommendations is set out below.

Value

  • We welcome the rapid development of these statistics and endorse your approach of adapting existing National Statistics data sources in combination with operational data. These statistics are used by the Cabinet Office to show the change in transport trends across Great Britain and give an indication of compliance with social distancing rules. They are vital for informing the government, the media and the general public and will continue to be valuable as the lockdown rules change and we enter the restart and recovery phase of the pandemic. We acknowledge the close engagement that your statisticians have on a regular basis with the Cabinet Office to ensure that the data are well understood. We also acknowledge and encourage your commitment to keep the statistics under review as the demand for new insights changes over the coming weeks and months.

Trustworthiness

  • The production of these statistics involves a coordinated and considerable effort across multiple analytical teams, overseen by yourself as Head of Profession for Statistics. Clear lines of accountability and responsibility have been established and resources have been reprioritised quickly to enable people to work on this. These measures contribute to demonstrating trustworthiness in the statistics.
  • However, these statistics are currently being released by the Cabinet Office if the information is included in the slides used in the daily briefing and not released if not. The Code of Practice for Statistics requires that official statistics should be released in an orderly and timely way at intervals that meet the needs of users as far as practicable. Given their public value, we regret that these statistics have not yet been consistently published on a regular basis. The Department for Transport should commit to publication at a specified time – say noon or 2pm – and on specified days, in a similar way to other COVID-related data such as the Public Health England deaths figures.

Quality

  • You have published a methodology note to accompany these statistics which clearly explains the data sources used, as well as any limitations or data quality issues and how these impact the trends presented. It is reassuring that where possible, you have used the same methodology for these statistics as is used in the related National Statistics publication and, where necessary, you have adapted the methodology to mitigate data quality issues related to the near-to real-time reporting period. Based on the availability of data, you have taken a practical approach to the choice of baselines to show the change in transport use over time.
  • You have implemented a thorough and robust quality assurance process to ensure that these statistics are high quality and mitigate the risk of error in the production process. This involves several levels of checks throughout the process and a range of quality assurance methods including comparisons to alternative data sources, use of industry intelligence and sense checks of changes in figures over time.
  • We welcome your commitment to continually improve your published methodology and quality information, for example by including more detailed information about Transport for London data and an explanation of the reason for different baselines used for different modes of transport.

We look forward to seeing these statistics develop as circumstances change. As set out in our guidance on changes to statistical outputs you can include a statement in your methodology note such as “These statistics have been produced quickly in response to developing world events. The Office for Statistics Regulation, on behalf of the UK Statistics Authority, has reviewed them against several key aspects of the Code of Practice for Statistics and regards them as consistent with the Code’s pillars of Trustworthiness, Quality and Value.”

I am copying this letter to Julie Stanborough, Deputy Director of the COVID-19 Press Data Team at the Cabinet Office.

Yours sincerely

 

Ed Humpherson

Director General for Regulation

 

Response: National Travel Survey – compliance check update

Dear Ian

Thank you for your letter regarding the steps taken by the National Travel Survey (NTS) team following our compliance check of the NTS statistics in 2018. I would like to thank you and your team for your proactive engagement and for the steps you have taken to improve and develop the NTS statistics.

I particularly welcome the innovative approaches taken by the team to improve the NTS statistics for users and the transparent presentation of this work, as well as the team’s collaboration with expert topic analysts to enhance the value and insights from the NTS. These initiatives really help to demonstrate the breadth and depth of NTS statistics. I would also like to highlight the accessible publication of users’ bespoke NTS ad-hoc requests to enable the reuse of this data by others. I’m really pleased to hear that this has resulted in further interest in the NTS statistics from new NTS users and further collaborative work between them and DfT statisticians.

Thank you again for your positive engagement and that of your team during this process. If you have any further questions regarding the future development of these statistics, please let me know.

Yours sincerely

Mary Gregory
Deputy Director for Regulation

 

Related Links:

Ian Knowles to Mary Gregory (May 2019)

Compliance Check – National Travel Survey (September 2018)

Compliance Check of Road Accidents and Safety Statistics

Dear Ian

ROAD ACCIDENTS AND SAFETY STATISTICS, GREAT BRITAIN

As you are aware, we have recently been reviewing your Road Accidents and Safety statistics against the Code of Practice for Statistics. We appreciate the positive engagement and clear commitment to the development of these statistics that your team showed during this process. I am pleased to confirm that these statistics should continue to be designated as National Statistics.

We have concluded that the Road Accidents and Safety statistics are produced independently and offer useful insights to users. Strengths of these statistics include:

  • the relevance and usefulness of these statistics to users – they are presented clearly, explained meaningfully and provide helpful insights that serve the public good;
  • the transparent descriptions of the quality limitations of the statistics, including the use of other data sources to triangulate the statistics and to understand the extent and nature of underreporting of road accidents;
  • the engagement with core users and user groups in terms of the future developments of the statistics;
  • the knowledge and professionalism of your team in advising users on appropriate use, engaging and listening to users’ needs, and handling data and methodology queries.

In the light of known limitations of the data sources and methods used to produce the statistics, we welcome your endeavours to develop a new adjusted injury severity series, and your engagement with the ONS Methodology Advisory Service to develop the new methods. This is a positive step to overcome the known quality limitations of the historical injury severity series for serious and slight injuries, and to enable you to provide better and clearer insights into road accidents. We also welcome your ambitions to develop consistent series in the light of two important changes that affect data collection – police forces adopting the CRASH system, which is a key step in improving quality and timeliness, and the phased introduction of online reporting of injuries by the public – both of which may affect the comparability of the data over time.

The Code of Practice for Statistics encourages innovation and development to ensure that statistics remain relevant to users in a changing world. Publication of new statistical series as experimental statistics gives a clear indication of such developments and is a positive endorsement of compliance with the Code’s practices around improvement and innovation. We therefore welcome your decision to clearly label the new adjusted injury severity series as experimental statistics,

signalling that they are undergoing evaluation and may be subject to further development following feedback from users.

We also welcome your ongoing commitment to further developing these statistics, including the planned improvements, which include:

  • a new flexible data dissemination tool which allows users to access data which meet their specific requirements in a timelier and less burdensome way;
  • publishing ad hoc analysis as standard, to enable reuse;
  • an assessment of the initial impact of the introduction of online reporting of injuries by the public.

I welcome these improvements and encourage your team to continue to be transparent around their future plans and timescales for the development of these statistics. This includes keeping users informed about progress with the workstreams currently being taken forward as part the broader STATS19 review, so that users may contribute their views and input into their evaluation. We would also encourage further developments in areas such as:

  • reviewing the quality of the data used to produce and quality assure the statistics and updating the published quality documentation;
  • reviewing users’ continued need for road accidents and safety statistics on a quarterly basis since production moved to mid and end year estimates temporarily from the end of 2018;
  • reviewing pre-release access to these statistics, with a view to keeping the number of individuals granted access to a minimum.

We will continue – through our new domain lead, Anna Price – to keep abreast of this work and will continue to reflect upon what its outcomes mean for the appropriate labelling of the serious and slight injury severity series going forwards.

I am copying this letter to Delphine Robineau, Head of Road Safety Statistics, Department for Transport.

Yours sincerely

Mary Gregory
Deputy Director for Regulation

National Travel Survey – compliance check update

Dear Mary

Response to the National Travel Survey Compliance Check

Thank you for your letter about the National Travel Survey (NTS) Compliance Check sent on 17 September 2018. I am formally responding as Head of Profession for Statistics to outline the steps we have taken to address the recommendations in the letter. Overall, we noted the letter was very positive about the survey and the engagement of the NTS lead statistician Darren Stillwell and his team with your team at the Office for Statistics Regulation. We were also pleased that the innovative work of the NTS team has subsequently been recognised with a case study on the Code of Practice website and a blog on the GSS website.

We had already addressed a number of recommendations as part of the July 2018 NTS statistical release. Since then, the team have been working towards addressing the final three recommendations in your letter. Darren emailed Oliver on 31 January 2019 with an informal update on progress and I am writing now to confirm the completion of all of three recommendations. I address each one below, along with the actions we have taken:

A new NTS quality report, capturing existing and internal quality management approaches, confidentiality protection, and updated standard errors and confidence intervals.

In January 2019 the NTS team published a user friendly quality report published covering those aspects of quality, and others.

At the same time, we also published a set of standard errors and confidence interval tables updated for 2017. We also have a new process now to produce more confidence intervals for our 2018 release, due in July 2019.

A new system for publishing ad hoc NTS analysis as standard, to enable reuse.

In January 2019 we published external 34 ad hoc requests, and have a process now in place to do this at least once a year, which will be alongside one or both of our two main NTS statistical publications. We have had many follow up queries on these tables and this has highlighted the benefit of publishing them.

Publishing information on users views and experiences of the NTS

On 29 May 2019, we published four documents:

a) A Discovery report investigating the benefits of designing a digital diary for the NTS;
b) A Discovery report investigating the benefits of designing an interactive analysis tool for the NTS;
c) An NTS user feedback exercise asking for views on proposed changes to survey questions; and
d) A proposed revised questionnaire to sit alongside the user feedback document.

These are the first four documents here. The Discovery reports detail a wide range of user views about the National Travel Survey. They outline how respondents, interviewers, NTS contractors, and internal and external end users of the data might benefit from either a self-service online analysis tool or a digital travel diary.

Further to this, I would also like to draw your attention to our continued user engagement and transparency with some other publications. In January 2019 we published:

a) Articles from external users showing innovative uses of the NTS data. This was the second in the series after we introduced the concept in January 2018.

b) Two NatCen reports on testing questions, and incentive and advance letter experiments: one and two. Ultimately, these experiments were unsuccessful in that they did not show an improvement in response rates for the survey. However, we feel it is important to publish these methodological reports to continue to show how we are continually striving to improve the NTS.

Finally, in May 2019 we published the results of the first wave of our new NTS panel survey, the National Travel Attitudes Study. This first wave consisted of questions previously asked on the British Social Attitudes Survey, but also 17 new questions driven entirely by policy needs within the Department on important topics such as low emission vehicles and seatbelt use. It will provide a flexible way of addressing policy needs for evidence in the future.

I hope this letter addresses all the remaining points of your original letter, and please contact either me or Darren if you wish to discuss any aspects further.

Yours sincerely,

Ian Knowles
Head of Profession for Statistics, Department for Transport

 

Related Links:

Mary Gregory to Ian Knowles (October 2019)

Compliance Check – National Travel Survey (September 2018)