Reproducible Analytical Pipelines: Overcoming barriers to adoption

Executive Summary

Introduction to the review

Official statistics produced by governments should uphold the highest standards of trustworthiness, quality and value in order to serve the public good. In 2017 we championed the Reproducible Analytical Pipeline (RAP), a new way of producing official statistics developed by the Department for Culture, Media and Sport and the Government Digital Service. This approach involved using programming languages to automate manual processes, version control software to robustly manage code and code storage platforms to collaborate, facilitate peer review and publish analysis.

Since then, we have seen some excellent examples of RAP principles being applied across the Government Statistical Service (GSS), the cross-government network of all those who work on official statistics. However, through our regulatory work we have seen that there are often common barriers for teams and organisations wishing to implement RAP. These include access to the right tools and training and statisticians having the time and support to carry out development work.

In Summer 2020 we set out our intention to further advocate for RAP principles in government statistics as part of our Automation and Data programme. We consider that RAP principles support all three pillars of the Code of Practice for Statistics: trustworthiness, quality and value.

In Autumn 2020 we launched this review. Our aim was to explore the current use of RAP principles across the GSS, identify what enables successful implementation and to understand what prevents statistics producers implementing RAP. We spoke to a variety of organisations that produce official statistics. This included the Office for National Statistics, UK government departments, devolved administrations, arms-length-bodies and voluntary adopters of the Code of Practice for Statistics. We also engaged with users of official statistics and stakeholders with a supportive or leadership role in this area, such as the GSS Best Practice and Impact Team and the office of the National Statistician. Finally, we drew on other available sources of evidence. These included Civil Service and GSS surveys and findings from our previous regulatory work. More information about how we carried out the review is provided in Annex 1: Approach to the review.

Our findings and recommendations

To enhance the trustworthiness, quality and value of official statistics through increased use of RAP principles and see RAP become the default approach to statistics we make the following recommendations.

FindingRecommendation
A consistent shared understanding of RAP and RAP principles is needed across the GSS.Building on their previous work to promote RAP, the Best Practice and Impact Team and RAP champions network should ensure that there is widespread awareness within the GSS of the recently developed minimum standard of RAP.
RAP is not only a change in tools – it involves a cultural change to the way that analysis is approached and carried out.The Analysis Function board and Directors of Analysis should consider how best to foster a culture where reproducible analysis is prioritised across government.
RAP principles support the highest standards of trustworthiness, quality and value and should be used as a way to enhance compliance with the Code of Practice for Statistics.The leadership of the GSS, including the National Statistician, should set a strategic direction for the use of RAP principles in official statistics.
Support and encouragement from senior leaders allows statistics producers to successfully and sustainably implement RAP.Organisations in the GSS should ensure that RAP principles are included in their analytical strategies.
Senior leaders responsible for strategies in their organisations must develop a good understanding of what RAP is, why it is required, and support an open culture of innovation.
The implementation of RAP principles is most successful when producers carry out their own development work and when a planned approach is taken – for example having a good understanding of skill levels, training needs and existing processes.Statistics producers should take a managed approach to implementing RAP. Projects should be underpinned by senior support, sufficient resource and the required skills, training and mentoring support.
RAP is not all or nothing: implementing just some RAP principles will result in improvements.Statistics producers should consider what can be achieved easily and build on developments iteratively over time.
Programming and code management skills are essential for modern statistical analysis.The GSS People Committee should ensure that RAP-related skills such as coding and code management are considered core skills for statistics producers and included in future career frameworks, such as the competency framework.
Bespoke and targeted training is most successful. Statistics producers need access to advanced training on programming, as well as introductory courses.The GSS should invest in advanced and bespoke training on RAP and RAP-related skills through the Analytical Learning Team. This should build on existing resources and be developed in collaboration with the Best Practice and Impact Team. Availability of training must be effectively communicated across the GSS so everyone is aware of it.
Support from experts has a significant impact on the success of RAP projects.The GSS needs to invest in expert mentoring, for example, through the Best Practice and Impact Team. Organisations that have the required skills and knowledge should support those that don’t.
Access to the tools required for RAP, such as programming languages, version control software and code storage platforms, varies across organisations. Organisations are tackling the same technical problems with different results.A strategy for implementing RAP principles across the GSS should recommend tools which should be available to statistics producers. It should also provide guidance on the best approaches to solving common technical problems.

Statistical leadership: Making analytical insight count

Executive Summary

This report sets out the findings from our review of statistical leadership. It looks at how statistical leadership can be strengthened across government.

Strong statistical leadership is essential to ensuring statistics serve the public good. Many decisions draw on statistics published by governments across the UK. Successful implementation of government policies can be dependent on public confidence in the data and messages shared by government. Individuals need to be confident in the data and associated narratives in order to make decisions which impact on their lives, business, or charities.

Governments need to be role models for statistical leadership. They need statisticians who can show leadership within the profession and across their organisations, and officials who can champion the use of evidence and be confident in engaging with analytical experts. All those with public facing roles must be capable of communicating messages drawing on data to support public confidence in data and how they have been used.

The report is intended to act as a starting point for further engagement. We will be engaging widely across analytical and other professions and plan to provide a progress update to this report in 2022. If you have feedback or would like to discuss any aspects of this report please contact us.

Mental Health Statistics in England

Attitudes towards mental health have changed in recent years. Mental health, which was often stigmatised and not discussed openly, is receiving increasing public, media and government attention as an important public health issue. There is a greater awareness that mental health is something we all have and, just like physical health, it can sometimes be good and sometimes be poor.

Our review of mental health statistics in England, carried out before the Covid-19 pandemic, explores why good statistics in this area are important, but is not intended to provide specific guidance on statistics directly related to the effects of the pandemic. We hope however, that sharing our findings on the strengths and weaknesses of mental health statistics, along with highlighting specific recommendations for improvements, will help inform decisions in the statistical sector both in the immediate term and going forward.

Our research for this review focused on answering the following two questions:

  • is the mental health statistical system publishing the information required to provide individuals, service providers and policy makers with a comprehensive picture on mental health?
  • do the existing statistics help answer the key questions about mental health in society today?

We spoke to a wide range of statistics users across different areas of society. They told us of their need for high quality statistics which are able to answer a broad range of questions. Users told us that the existing statistics did not paint a full enough picture of individuals and their conditions, and that producers should be taking greater steps to maximise the insight from existing statistics. In some areas they wanted to know more than the current statistics were able to tell them.

We heard that there is a need for improved quality across the datasets underlying many mental health statistics. Users told us that mental health statistics should be more accessible, both in terms of finding relevant publications and in relation to producers making publications easy to read and explaining clearly the limitations of the statistics. In addition to this, they spoke of their frustrations that some surveys were not carried out as often as they would like, as well as challenges around obtaining data for secondary analysis purposes.

Our research identified that, although the existing mental health statistics go some way to meeting user’s needs, there is much more that can be done.

Our recommendations:

  1. Statistics producers and organisations should exploit the value of the statistics through better data, greater analysis and linking data.
  2. We want to see continued activity to improve the quality of underlying statistics datasets, as well as clear communication with users about quality issues.
  3. We want to see clearer leadership and greater collaboration across producers of mental health statistics.
  4. Access to NHS Digital data needs to improve.

We understand that addressing these issues may not currently be a priority for statistics producers due to the COVID-19 situation, however we expect statistics producers to work collaboratively towards delivering these recommendations when they are able to do so.

 

Related Links:

Chris Roebuck to Ed Humpherson: NHS Digital update on mental health statistics