Reproducible Analytical Pipelines: Overcoming barriers to adoption

Executive Summary

Introduction to the review

Official statistics produced by governments should uphold the highest standards of trustworthiness, quality and value in order to serve the public good. In 2017 we championed the Reproducible Analytical Pipeline (RAP), a new way of producing official statistics developed by the Department for Culture, Media and Sport and the Government Digital Service. This approach involved using programming languages to automate manual processes, version control software to robustly manage code and code storage platforms to collaborate, facilitate peer review and publish analysis.

Since then, we have seen some excellent examples of RAP principles being applied across the Government Statistical Service (GSS), the cross-government network of all those who work on official statistics. However, through our regulatory work we have seen that there are often common barriers for teams and organisations wishing to implement RAP. These include access to the right tools and training and statisticians having the time and support to carry out development work.

In Summer 2020 we set out our intention to further advocate for RAP principles in government statistics as part of our Automation and Data programme. We consider that RAP principles support all three pillars of the Code of Practice for Statistics: trustworthiness, quality and value.

In Autumn 2020 we launched this review. Our aim was to explore the current use of RAP principles across the GSS, identify what enables successful implementation and to understand what prevents statistics producers implementing RAP. We spoke to a variety of organisations that produce official statistics. This included the Office for National Statistics, UK government departments, devolved administrations, arms-length-bodies and voluntary adopters of the Code of Practice for Statistics. We also engaged with users of official statistics and stakeholders with a supportive or leadership role in this area, such as the GSS Best Practice and Impact Team and the office of the National Statistician. Finally, we drew on other available sources of evidence. These included Civil Service and GSS surveys and findings from our previous regulatory work. More information about how we carried out the review is provided in Annex 1: Approach to the review.

Our findings and recommendations

To enhance the trustworthiness, quality and value of official statistics through increased use of RAP principles and see RAP become the default approach to statistics we make the following recommendations.

FindingRecommendation
A consistent shared understanding of RAP and RAP principles is needed across the GSS.Building on their previous work to promote RAP, the Best Practice and Impact Team and RAP champions network should ensure that there is widespread awareness within the GSS of the recently developed minimum standard of RAP.
RAP is not only a change in tools – it involves a cultural change to the way that analysis is approached and carried out.The Analysis Function board and Directors of Analysis should consider how best to foster a culture where reproducible analysis is prioritised across government.
RAP principles support the highest standards of trustworthiness, quality and value and should be used as a way to enhance compliance with the Code of Practice for Statistics.The leadership of the GSS, including the National Statistician, should set a strategic direction for the use of RAP principles in official statistics.
Support and encouragement from senior leaders allows statistics producers to successfully and sustainably implement RAP.Organisations in the GSS should ensure that RAP principles are included in their analytical strategies.
Senior leaders responsible for strategies in their organisations must develop a good understanding of what RAP is, why it is required, and support an open culture of innovation.
The implementation of RAP principles is most successful when producers carry out their own development work and when a planned approach is taken – for example having a good understanding of skill levels, training needs and existing processes.Statistics producers should take a managed approach to implementing RAP. Projects should be underpinned by senior support, sufficient resource and the required skills, training and mentoring support.
RAP is not all or nothing: implementing just some RAP principles will result in improvements.Statistics producers should consider what can be achieved easily and build on developments iteratively over time.
Programming and code management skills are essential for modern statistical analysis.The GSS People Committee should ensure that RAP-related skills such as coding and code management are considered core skills for statistics producers and included in future career frameworks, such as the competency framework.
Bespoke and targeted training is most successful. Statistics producers need access to advanced training on programming, as well as introductory courses.The GSS should invest in advanced and bespoke training on RAP and RAP-related skills through the Analytical Learning Team. This should build on existing resources and be developed in collaboration with the Best Practice and Impact Team. Availability of training must be effectively communicated across the GSS so everyone is aware of it.
Support from experts has a significant impact on the success of RAP projects.The GSS needs to invest in expert mentoring, for example, through the Best Practice and Impact Team. Organisations that have the required skills and knowledge should support those that don’t.
Access to the tools required for RAP, such as programming languages, version control software and code storage platforms, varies across organisations. Organisations are tackling the same technical problems with different results.A strategy for implementing RAP principles across the GSS should recommend tools which should be available to statistics producers. It should also provide guidance on the best approaches to solving common technical problems.

Statistical leadership: Making analytical insight count

Executive Summary

This report sets out the findings from our review of statistical leadership. It looks at how statistical leadership can be strengthened across government.

Strong statistical leadership is essential to ensuring statistics serve the public good. Many decisions draw on statistics published by governments across the UK. Successful implementation of government policies can be dependent on public confidence in the data and messages shared by government. Individuals need to be confident in the data and associated narratives in order to make decisions which impact on their lives, business, or charities.

Governments need to be role models for statistical leadership. They need statisticians who can show leadership within the profession and across their organisations, and officials who can champion the use of evidence and be confident in engaging with analytical experts. All those with public facing roles must be capable of communicating messages drawing on data to support public confidence in data and how they have been used.

The report is intended to act as a starting point for further engagement. We will be engaging widely across analytical and other professions and plan to provide a progress update to this report in 2022. If you have feedback or would like to discuss any aspects of this report please contact us.

Mental Health Statistics in England

Attitudes towards mental health have changed in recent years. Mental health, which was often stigmatised and not discussed openly, is receiving increasing public, media and government attention as an important public health issue. There is a greater awareness that mental health is something we all have and, just like physical health, it can sometimes be good and sometimes be poor.

Our review of mental health statistics in England, carried out before the Covid-19 pandemic, explores why good statistics in this area are important, but is not intended to provide specific guidance on statistics directly related to the effects of the pandemic. We hope however, that sharing our findings on the strengths and weaknesses of mental health statistics, along with highlighting specific recommendations for improvements, will help inform decisions in the statistical sector both in the immediate term and going forward.

Our research for this review focused on answering the following two questions:

  • is the mental health statistical system publishing the information required to provide individuals, service providers and policy makers with a comprehensive picture on mental health?
  • do the existing statistics help answer the key questions about mental health in society today?

We spoke to a wide range of statistics users across different areas of society. They told us of their need for high quality statistics which are able to answer a broad range of questions. Users told us that the existing statistics did not paint a full enough picture of individuals and their conditions, and that producers should be taking greater steps to maximise the insight from existing statistics. In some areas they wanted to know more than the current statistics were able to tell them.

We heard that there is a need for improved quality across the datasets underlying many mental health statistics. Users told us that mental health statistics should be more accessible, both in terms of finding relevant publications and in relation to producers making publications easy to read and explaining clearly the limitations of the statistics. In addition to this, they spoke of their frustrations that some surveys were not carried out as often as they would like, as well as challenges around obtaining data for secondary analysis purposes.

Our research identified that, although the existing mental health statistics go some way to meeting user’s needs, there is much more that can be done.

Our recommendations:

  1. Statistics producers and organisations should exploit the value of the statistics through better data, greater analysis and linking data.
  2. We want to see continued activity to improve the quality of underlying statistics datasets, as well as clear communication with users about quality issues.
  3. We want to see clearer leadership and greater collaboration across producers of mental health statistics.
  4. Access to NHS Digital data needs to improve.

We understand that addressing these issues may not currently be a priority for statistics producers due to the COVID-19 situation, however we expect statistics producers to work collaboratively towards delivering these recommendations when they are able to do so.

Exploring the public value of statistics about post-16 education and skills – UK report

We have been looking in detail at the value of the current data and statistics on post-16 education and skills. As an independent UK wide regulator, we are in a unique position to take a broader look at issues of importance to society and to make the case for improved statistics, across organisational and Government boundaries. 

This report, our second report in this topic area, explores the public value of post-16 education and skills statistics across the UK with a focus on Scotland, Wales and Northern Ireland and updates on changes since the publication of our first, England only, report in 2019. 

Four key sectors comprise the majority of the post 16 education and skills statistics in the UK: workforce skills, universities and higher education, colleges and further education and apprenticeships, and each are covered in detail in our report. To our knowledge, this is the first time that the statistics that inform these sectors have been extensively researched at a UK wide level.  

Exploring the statistical landscape in this multi sector, multi country way has allowed us, to not only to identify the current challenges, information gaps and improvements to statistics in each sector, but to also highlight areas of good practice and shared learning opportunities. We have looked in detail as to how the current statistics are meeting the needs of users, focusing on the public value that the statistics give. In doing this we have been also been able to explore in detail how accessible the current statistics are and whether theare helping to inform a bigger, sector wide, picture. 

Post-16 education and skills affect the lives of millions of individuals in the UK. Good quality and accessible statistics are important to support the fair, efficient and effective provision of education and training. Alongside this report we will continue to engage with statistics producers to make the case for improved data and statistics in these sectors 

The state of the UK’s statistical system

This review sets out our view on the current state of government statistics. At their best, statistics and data produced by government are insightful, coherent, and timely. They are of high policy-relevance and public interest. There are good examples of statistics that effectively support decision-making in many areas of everyday life: this has been especially true during the COVID-19 pandemic, when we’re seeing the kind of statistical system that we’ve always wanted to encourage – responsive, agile and focusing on users. However, the statistical system does not consistently perform at this level across all its work.

In this report we address eight key areas where improvements could be made across the system.

  1. Statistical leadership
  2. Voluntary Application of the Code, beyond official statistics
  3. Quality assurance of administrative data
  4. Communicating uncertainty
  5. Adopting new tools, methods and data sources
  6. Telling fuller stories with data
  7. Providing authoritative insight
  8. User engagement

In each area, we highlight examples of statistical producers doing things well. These examples illustrate the good work already happening which others can learn from and build on. We have organised our reflections under the three headings of Trustworthiness, Quality and Value, the three essential pillars that provide the framework for the Code of Practice for Statistics.

User engagement in the Defra Group

Why we did this review

Understanding how statistics are used and what users and other stakeholders need is critical to ensuring that statistics remain relevant and provide insight. To achieve this, statistics producers must engage with users.

To explore this aspect of statistics production, we carried out a review of user engagement in the Defra Group. By the Defra group we mean the Core Department and Executive Agencies, Forestry Commission and those Defra Arm’s Length bodies that are designated as producers of official statistics: Environment Agency, Joint Nature Conservation Committee, Marine Management Organisation and Natural England.

This is our first departmental review of user engagement and the Defra Group made an ideal candidate for such a review. It has a large and broad portfolio of official statistics and National Statistics, with a varied public profile, public interest and impact and is therefore likely to require different approaches to engaging with users.

We focused our review on a set of 10 National Statistics and official statistics which reflect the diversity of the Defra Group statistics portfolio (see report Annex B). They cover a range of topics, users and uses, and represent the Defra core department as well as Arm’s Length Bodies.

What we hope to achieve

Through this review we aim to develop a better understanding of the range of approaches to user engagement currently adopted within the Defra Group, and to identify the key features of effective and impactful user engagement. We hope this will support the Defra Group in enhancing its user engagement and provide broader learning for other statistics producers.

Related links:

Correspondence: Ken Roy to Ed Humpherson: User engagement in the Defra group

Correspondence: Ed Humpherson to Ken Roy: User engagement in the Defra Group

Blog: What we have learned from the Defra Group about user engagement

Strengthening the quality of HMRC’s official statistics

Introduction to the review

In September 2019, HMRC invited the Office for Statistics Regulation to carry out a review of the principles and processes underpinning the quality of HMRC’s official statistics. This review was proactively initiated after HMRC identified a significant error in published Corporation Tax receipt statistics, which affected the period from April 2011 to July 2019.

Aim and scope of the review

The aim of our review was to provide an independent assessment of the approach that HMRC takes to manage quality and risk in the production of its official statistics and to identify potential improvements. We appreciate that producers of statistics will never eliminate errors entirely: the recommendations we present in this report focus on improvements that HMRC should make to help minimise the risk of issues with its statistics in the future.

Related links:

Ed Humpherson to Ruth Stanier: Strengthening the quality of HMRC’s official statistics

Ed Humpherson to Jim Harra: Strengthening the quality of HMRC’s official statistics

Ed Humpherson to Sean Whellams: Review of HMRC statistical quality management

Jim Harra to Ed Humpherson

Ruth Stanier to Ed Humpherson 

Adult Social Care Statistics: Summary Report for Great Britain

Following the publication of our reports on adult Social Care statistics for England, Scotland and Wales, the Office for Statistics Regulation has published a summary report for Great Britain.

This report draws together the main findings from each of the three countries. We closed the project strand for Northern Ireland, following the publication of a letter in March 2019.

We spoke to a range of users of these statistics, as well as reviewing existing outputs. Given the devolved nature of adult social care, we looked at statistical issues in each of the four countries separately.

Today’s report highlights common challenges and frustrations, as well as good practice relating to adult social care. It concludes with our priorities for action that each of the three countries should take to improve adult social care data and statistics.

Related Links:

Report on Adult Social Care statistics in England (January 2020)

Adult Social Care Statistics in Scotland (February 2020)

Adult Social Care Statistics in Wales (June 2019)

Two-year update: Public Value of Statistics on Housing and Planning in the UK

Our systemic review of The Public Value of Statistics on Housing and Planning in the UK was published in November 2017. This comprehensive review looked across a wide range of the statistics within our Housing, Planning and Local Services regulatory area.

This two-year update report shares the progress made since the review, highlights the challenges that remain and outlines our proposed work plan approach for this regulatory area.

 

Related Links

Public value of Statistics on Housing and Planning in the UK (November 2017)