Transparency: bringing the inside out

In our latest blog Director General for Regulation, Ed Humpherson, discusses the divergence between internal positivity and external scepticism about analysis in Government, and how transparency is key to benefitting the public good…

Seen from within Government, these are positive times for analysis. There is an analysis function, headed by Sir Ian Diamond, which continues to support great, high-profile analytical work. There are strong professions, including economists, statisticians, operational researchers and social researchers, each with strong methods and clear professional standards. There is an Evaluation Task Force, which is doing great things to raise the profile of evaluation of policy. And data and analysis are emphasised by Ministers and civil service leaders like never before – exemplified by the 2023 One Big Thing training event focused on use of data and analysis in Government.

Yet the perspective from outside Government is quite different. The Public Administration and Constitutional Affairs Select Committee has been undertaking an inquiry into Transforming the UK’s Statistical Evidence Base. Several witnesses from outside Government who’ve given evidence, and some of the written evidence that has been provided, highlights concerns about the availability of analysis and how it’s used. In particular, witnesses questioned whether it’s clear what evidence sources inform policy decisions.

What explains this divergence between internal positivity and external scepticism?

In my view, and as I said in my own evidence before the Committee, it all comes down to transparency. By this I mean: the way in which analysis, undertaken by officials to inform Ministers, is made available to external users.

This is highly relevant to the Committee’s inquiry. A key question within the inquiry is the way in which external users can access analysis undertaken within Government.

These questions are very relevant to us in OSR. We have developed the principle of Intelligent Transparency. You can read more here, but in essence, Intelligent Transparency is about ensuring that, when Government makes statements using numbers to explain a policy and its implementation, it should make the underlying analysis available for all to see.

As I explained to the Committee, we make interventions when we see this principle not being upheld – for example, here and here. When we step in departments always respond positively, and the analysts work with policy and communications colleagues to make the evidence available.

My basic proposition to the Committee was that the more Government can comply with this principle, the more the gap between the internal insight (there’s lots of good analysis) and the external perception (the analysis isn’t used or made available), will close. This commitment to transparency should be accompanied by openness – the  willingness to answer questions raised by users; and a willingness to acknowledge the inherent limitations and uncertainties within a dataset.

In terms of what we do at OSR, I wouldn’t see any point, or value, in us going upstream to consider the quality of all the analysis that circulates within Government.

Our role is about public accessibility and public confidence – not about an internal quality assurance mechanism for economics, operational research, social research and other types of analysis undertaken in Government. We are not auditors of specific numbers (ie a particular figure from within a statistical series) – something we have to reiterate from time to time when a specific number becomes the focus of political debate. Nor do we have the resources nor remit to do that. But we DO have both the capacity and framework to be able to support the appropriate, transparent release and communication of quantitative information.

This is the heartland of our work on statistics, and it’s completely applicable to, say, economic analysis of policy impacts, or evaluations of the impact of Government policy. There are good arrangements for the quality of economic analyses through the Government Economic Service (GES), and the quality of evaluations through the Evaluation Task Force (ETF); and similarly for the other disciplines that make up the Analysis Function. The ETF is a new kid on this particular block, and it is a great innovation, a new force for driving up the standards and openness of Government evaluations.

Where we add value is not in duplicating the GES, or ETF, or similar professional support structure within Government. Indeed, we already work in partnership with these sources of support and professional standards. Our expertise is in how this quantitative information is communicated in a way that can command public confidence.

In short, then, it really does come down to a question of transparency. As I said to the Committee, it’s like a garden in the early morning. Some of it is in the sunlight already, and some of it still in shade. Gradually, we are seeing more and more of the lawn come into the sunlight – as the reach of transparency grows to the benefit of the public.

The success and potential evolution of the 5 Safes model of data access

In our latest blog Ed Humpherson, Director General for Regulation discusses the 5 Safes model as a key feature to support data sharing and linkage…


In OSR’s data linkage report , we highlighted the key features of the data landscape that support data sharing and linkage. The 5 Safes model is one of those. Yet we also recommended that the 5 Safes model is reviewed. In this blog, I want to focus on one aspect of the model and set out the case for a subtle but important change.

The 5 Safes model is an approach to data use that has been adopted widely across the UK research community, and has also been used internationally. It is well-known and well-supported and has had a significant impact on data governance. It is, in short, a huge success story. (And for a short history, and really interesting analysis, see this journal article by Felix Ritchie and Elizabeth Green).

The 5 Safes are:

  • Safe data: data is treated to protect any confidentiality concerns.
  • Safe projects: research projects are approved by data owners for the public good.
  • Safe people: researchers are trained and authorised to use data safely.
  • Safe settings: a SecureLab environment prevents unauthorised use.
  • Safe outputs: screened and approved outputs that are non-disclosive.

Any project that aims to use public sector administrative data for research purposes should be considered against the 5 Safes. The 5 Safes therefore is used to set a criteria-based framework for providing assurance about the appropriateness of a particular project.

OSR’s recommendations relevant to the 5 Safes:

In July 2023, OSR published our report on data sharing and linkage in government. We had a range of findings. I won’t spell them out here, but in short, we found a good deal of progress across Government, but some remaining barriers to data sharing and linkage. We argued that these barriers must be addressed to ensure that the good progress is maintained.

We made two recommendations relevant to the 5 Safes:

  • Recommendation 3: The Five Safes Framework Since the Five Safes Framework was developed twenty years ago, new technologies to share and link data have been introduced and data linkage of increased complexity is occurring. As the Five Safes Framework is so widely used across data access platforms, we recommend that the UK Statistics Authority review the framework to consider whether there are any elements or supporting material that could be usefully updated.
  • Recommendation 10: Broader use cases for data To support re-use of data where appropriate, those creating data sharing agreements should consider whether restricting data access to a specific use case is essential or whether researchers could be allowed to explore other beneficial use cases, aiming to broaden the use case were possible.

We made the recommendation about reviewing the framework because a range of stakeholders mentioned to us the potential for updating the 5 Safes model, in the light of an environment of ever-increasing data availability and ever-more powerful data processing and analysis tools.

And we made the recommendation about broader use cases because this was raised with us as an area of potential improvement.

The use of 5 Safes in research projects

What brings the two recommendations together is the 5 Safes idea of “safe projects”. This aspect of the model requires research projects to be approved by data owners (essentially, the organisations that collect and process the data) for the public good.

For many research activities, this project focus is absolutely ideal. It can identify how a project serves the public good, what benefits it is aiming to bring, and any risks it may entail. It will require the researcher to set out the variables in the data they wish to explore, and the relationships between those variables they want to test.

For some types of research, however, the strictures of focusing on a specific project can be limiting. For example, for a researcher who wants to establish a link between wealth and some aspects of health may not know in advance which of the variables in a wealth dataset, and which of the variables in a health data set, they wish to examine. Using the “safe project” framing, they might have to set out specific variables, only to discover that they are not the most relevant for their research. And then they might have to go back to the drawing board, seeking “safe project” approval for a different set of variables.

Our tentative suggestion is that a small change in focus might resolve these problems. If the approval processes focused on safe programmes, this would allow approval of a broad area of research – health and wealth data sets – without the painstaking need to renew applications for different variables within those datasets.

What I have set out here is, of course, very high level. It would need quite a lot of refinement.

Other expert views on the 5 Safes

Recognising this, I shared the idea with several people who’ve spent longer than me thinking about these issues. The points they made included:

  • Be careful about placing too much emphasis on the semantic difference between programmes and projects. What is a programme for one organisation or research group might be a project for another. More important is to establish clearly that broader research questions can be “safe”. Indeed, in the pandemic, projects on Covid analysis and on Local Spaces did go ahead with a broader-based question at their heart.
  • This approach could be enhanced if Data Owners and Controllers are proactive in setting out what they consider to be safe and unsafe uses of data. For example, they could publish any hard-line restrictions (“we won’t approve programmes unless they have the following criteria…”). Setting out hard lines might also help Data Owners and Controllers think about programmes of research rather than individual projects by focusing their attention on broader topics rather than specifics.
  • In addition, broadening the Safe Project criterion is not the only way to make it easier for researchers to develop their projects. Better meta data (which describe the characteristics of the data) and synthetic data (which create replicas of the data set) can also help researchers clarify their research focus without needing to go through the approvals process. There have already been some innovations in this area – for example, the Secure Research Service developed an exploratory route that allows researchers to access data before putting in a full research proposal – although it’s not clear to me how widely this option is taken up.
  • Another expert pointed out the importance of organisations that hold data being clear about what’s available. The MoJ Data First programme provides a good example of what can be achieved in this space – if you go to the Ministry of Justice: Data First – GOV.UK (www.gov.uk) you can see the data available in the Datasets section, including detailed information about what is in the data.
  • Professor Felix Ritchie of the University of West England, who has written extensively about data governance and the 5 safes, highlighted for me that he sees increasing “well-intentioned, but poorly thought-through” pressure to prescribe research as tightly as possible. His work for the ESRC Future Data Services project sees a shift away from micro-managed projects as highly beneficial – after all, under the current model “the time risk to a researcher of needing a project variation strongly incentivises them to maximise the data request”.

More broadly, the senior leaders who are driving the ONS’s Integrated Data Service pointed out that the 5 Safes should not be seen as separate minimum standards. To a large extent, they should be seen as a set of controls that work in combination – the image of a graphic equaliser to balance the sound quality in a sound system is often given. Any shift to Safe Programmes should be seen in this context – as part of a comprehensive approach to data governance.

Let us know your thoughts

In short, there seems to be scope for exploring this idea further. Indeed, when I floated this idea as part of my keynote speech at the ADR UK conference in November, I got – well, not quite a rapturous reception, but at least some positive feedback.

And even if it’s a small change, of just one word, it is nevertheless a significant step to amend such a well-known and effective framework. So I offer up this suggestion as a starter for debate, as opposed to a concrete proposal for consultation.

Let me know what you think by contacting DG.Regulation@Statistics.gov.uk.

Producing, reviewing, and always evolving: UKHSA statistics

In our latest guest blog Helen Barugh, Head of Statistics Policy and User Engagement, discusses transforming the statistics produced by the UK Health Security Agency…

 

What is the Health Security Agency?

The UK Health Security Agency (UKHSA) is responsible for protecting every member of every community from the impact of infectious diseases, chemical, biological, radiological and nuclear incidents and other health threats. We are an executive agency of the Department for Health and Social Care.

We collect a wide range of surveillance data about diseases ranging from influenza and COVID-19 to E.coli to measles. We publish statistics related to planning, preventing and responding to external health threats. You can find our statistics here.

UKHSA was born in October 2021 during the COVID-19 pandemic. The pandemic had a significant impact on the organisation, including statistical production, and the repercussions of that are still being felt.

Producing and reviewing official statistics

I joined UKHSA in August 2022, one of four recruits to a new division supporting the statistics head of profession. Our division, which has a mix of statisticians at different grades, aims to transform UKHSA statistical production and dissemination. We have all produced statistics in other government departments, and we use that expertise to provide advice, guidance and practical support to all aspects of statistics production and dissemination. Our division also includes two content designers who actually publish UKHSA’s official statistics.

One of the most important parts of our work is a programme of reviews looking in-depth at each UKHSA official statistics publication. This is a big programme of work, encompassing around 35 statistical series covering a range of topics and including weekly reports right through to annual reports. The reviews aim to:

  • bring consistency to our statistics production and outputs.
  • improve efficiency and quality assurance through the adoption of reproducible analytical pipelines (RAP) in line with our RAP implementation plan.
  • improve compliance with the Code of Practice for Statistics.
  • embed user engagement as a regular and standard activity.

We are part of the way through this programme, having reviewed around 20 series and with another 15 to go. We expect to finish the reviews by late summer 2024.

How do we review our statistics?

Our reviews have three main phases.

  • Desk-based research includes assessing products against the Code of Practice for Statistics, assessing publications for accessibility and clarity, reviewing desk notes and analysing google analytics to draw out insights about users.
  • Discovery work with the team explores the journey from data acquisition through to publication, understanding the processes used and the quality assurance in place. Sometimes we shadow a production cycle to really understand how the process works and how it can be improved. We also discuss user engagement with the team to investigate what they know about their users and how they assess any changing or emerging needs.
  • Once we have all the information we need, we write a report to summarise our findings and agree recommendations for improvement with the production team.

How do we make changes in practice?

We work with the production team, providing practical support, training and guidance as they implement the recommendations. We are aiming for incremental improvement, and the review provides a baseline against which we can measure success.

The reviews have given a terrific insight into the good practice within our organisation, as well as the challenges of producing some of our statistical products and the legacy processes that now need updating. Despite all the challenges of producing statistics during the pandemic, UKHSA statistics teams have been putting out very detailed and thorough statistics, in some cases on a weekly basis and with very short turnaround times between receiving data and publishing. Google analytics indicate that in general the readership is high, and user engagement so far has shown that products are highly valued and appreciated by users working in health protection and healthcare settings.

Areas for improvement are often similar across different statistical series. For example, production methods are not as reproducible as they could be. There are opportunities to introduce reproducible analytical pipelines (RAP) and build in automated quality assurance that will improve the efficiency and accuracy of production. We’ve also found that most outputs are aimed at a technical and clinical audience which limits their impact for the general public or for meeting the public good. As an organisation, we need to do more to understand the wider uses of our statistical products and adjust their presentation accordingly.

What difference are the reviews making?

One of the key benefits of the reviews has been building relationships between our division and UKHSA statistics teams. Our work only really has impact where teams get onboard and are enthused to make positive changes. So I’m delighted about the impact our work is having, with statistics teams working hard to improve their products. For example, charts are being redesigned to conform with best practice, RAP is being implemented in some publications, our first quality and methodology information report has been published and most publications are now in HTML. It all feels very positive!

So, what next?

We have more reviews to do to make sure we have an accurate picture of all our official statistics, and lots of opportunities to support production teams to make improvements. We’re planning more user engagement and also participating in the cross-government user consultation on health and social care statistics which will give us some valuable user feedback to help shape our statistics in the future.

We need to decide whether any publications designated on GOV.UK as ‘research and analysis’ should really be designated as official statistics. And as we bring about improvement right across the UKHSA statistics publications, we will be aiming for more products to be accredited by OSR to provide that stamp of approval that we’ve done the right things and are now meeting the highest standards of trustworthiness, quality and value.

If you’re interested in talking to us about our programme of reviews please do get in touch: UKHSA_HOPSTATS@ukhsa.gov.uk. We’re happy to share more about what we’ve learnt as well as the materials we’ve developed to support the review process.

You’re planning to do what? Statistics, resource constraints and user engagement

In our latest blog, Mark Pont, OSR’s Assessment Programme Lead and Philip Wales, NISRA’s Chief Executive discuss engaging with statistics users and how user input can help decision making… 

In his recent blog post about keeping a statistical portfolio (and a garden) sustainable, Rob Kent-Smith described some principles to consider when balancing scarce resources across a portfolio of statistics. In this post, Mark Pont, head of our compliance programme, brings this to life a little drawing on the recent experiences of NISRA – the Northern Ireland Statistics and Research Agency. Faced with some budgetary pressures, NISRA launched a consultation at the end of August 2023 and published a response just a few weeks ago.  

The Code of Practice for Statistics talks about the need to ensure that statistics remain relevant to users. The need for statistics producers to engage with users to understand their evolving needs is an important element of providing value.  

A myth that we sometimes hear at OSR is that accredited official statistics (the new name for what the Statistics and Registration Service Act 2007 calls National Statistics) can’t be stopped. We also hear that statistical outputs can only be added to. But it is important to recognise that just because a set of statistics has been produced in the past, this doesn’t mean that it must continue to be produced in the same way, with the same periodicity for evermore. Nor does the Code of Practice mandate a particular form of presentation or requirement for extensive commentary, as long as users’ needs are being met. To carry on the gardening idiom, sometimes plants need pruning or even removing to enable a garden to flourish. 

It’s therefore right that all options – reducing scope or frequency, or ceasing altogether – be considered.  

It’s also really important to recognise that a formal public consultation can form an important part of gathering users’ views. But this is best done within the context of more-proactive ongoing engagement, particularly with key decision-makers. 

It was therefore really good that Philip Wales, NISRA’s Chief Executive, contacted us to tell us about NISRA’s consultation. In the rest of this post, we ask Philip for some perspectives on how the consultation went, about how he went about engaging with users, and how their input helped his decision making.   

 

Mark: So Philip, first of all congratulations on the new role, which perhaps isn’t so new any more. How did it feel to be thrown straight into needing to make some tough decisions in the light of tight budgets? 

Philip: Thanks Mark – it’s been a challenging first ten months at NISRA, but I’ve really enjoyed it, and the time has flown by.  

You’re right to say that we faced – and continue to face – budgetary pressures at NISRA. Funding from our parent department will be around £1.9m lower in nominal terms this year compared to 2022-23. Because of inflation, that amounts to a real terms cut of close to 20% for our suite of economic, population and social statistics, not to mention our survey and data collection activities. 

To resolve this financial pressure, we’ve worked hard to find new sources of income and to move people into posts with dedicated funding, we’ve had to manage our resources well, and to think hard about our suite of outputs, which brings us to the consultation exercise we ran.  


Mark: How did you feel the consultation went? Was there anything that particularly pleased you about it, or its findings? 

Philip: The consultation we ran on our statistical outputs was an important part of managing our budgetary pressures. It gave us a chance to explain the financial context and communicate the pressures which NISRA is under to our users, and to talk about how we would manage them. It also encouraged us to think critically about  the work we do and where we add the greatest value.   

The consultation proposed changes to some of our planned outputs – either delaying them, scaling them back or suspending them – and enabled us to get feedback directly from our users.  

And on these terms, I think the consultation was a success. We had a large number of responses – from individuals, institutions, businesses and other organisations, as well as government departments – all of whom took the time to tell us that they really value the outputs we produce.  

From the feedback we got, we learned about where and how our releases are used, and we secured a better understanding of the potential impact of our proposed changes. Importantly, that feedback has helped to guide the changes we’re now making to our outputs.  


Mark: How has the consultation helped you to decide which activities and outputs to prioritise? And did you end up cutting back in the areas that you expected? 

Philip: The consultation helped us to work out how to minimise the impact of our proposals on our stakeholders.  

In lots of cases, users agreed that the changes we were proposing were the ‘least worst’ option available. Where we were combining outputs, or scaling them back to focus on the core headlines, users were understanding. Feedback also indicated that, in general, the outputs we were suspending were adding less value than our other activities: a sign we were focussing on the right things.  

Where we did meet real concern and resistance – particularly on some of our hospital infections releases and elements of our trade data suite – we listened. In these cases, we sought new and less resource intensive ways of meeting these needs.  

For me, this is the hallmark of a good consultation: asking people for feedback, listening, and then adjusting to account for their views.  


Mark: What were the most difficult parts of running the consultation? 

Philip: Well, it’ll be obvious that running a consultation like this – against a challenging financial background – isn’t a lot of fun!  

But I think the most difficult part of this process was the beginning. Sometimes, as producers, we can be a bit reluctant to ask the question ‘should I keep doing this?’ A bit like Rob said in his recent blog-post, it’s easy to think that an output should continue simply ‘because it has for a long time’, or ‘because it’s an Accredited Statistic’. People are protective of their outputs and can often be anxious when changes are discussed. 

I think this is a natural reaction, but it’s often an obstacle that we put in front of ourselves. The truth is that the skills of statisticians and analysts more broadly can be deployed in all kinds of really important ways across government, and we need to be thinking about how best to use those capabilities all the time. New datasets, new systems or new activities all mean there are new ways for skilled data analysts of all kinds to add value. In place of a guarded, defensive discussion, this lens really helped to promote the right kind of open discussion about outputs at NISRA.  


Mark: In conclusion, would you have any tips for others in a similar position? 

Philip: I think I’d give three short pieces of advice to someone undertaking an exercise like this one.  

Firstly, always keep in mind that making changes to an output isn’t a reflection on the people doing the work. Try and have an open, respectful conversation which captures the value that they can provide, recognising that sometimes less can be more.   

Second, trust and listen to one another. Changes like these are more likely to stick and less likely to have long term impacts on morale if it’s clear that you are doing this as part of a group.  

And third, trust your users. Listen to what they have to say and leave room to adjust your plans if you get unexpected feedback.  

If you want more advice about engaging with users as part of prioritisation exercises, please do contact us. 

NHS England guest blog: Mental health data quality

In our latest guest blog Gary Childs, Head of Analytical Delivery at NHS England, discusses steps NHS England has been taking in recent years to improve the quality of mental health data…

Context 

Within NHS England we have been striving to improve the quality of mental health data for a number of years. In this blog we will focus on the Mental Health Services Dataset (MHSDS), but the methodologies and principles apply equally to datasets reporting on topics such as NHS Talking Therapies or Learning Disabilities and Autism (LDA).  

Clinical Data in National Collections

Charles Babbage once said: “Errors using inadequate data are much less than those using no data at all”. However, there must be a threshold as poor-quality data can lead to inaccurate analytics, bad decisions, and in health, have an impact on patient care. The Government Data Quality Hub states that: “Good quality data is data that is fit for purpose. That means the data needs to be good enough to support the outcomes it is being used for”.  

In health, one would automatically assume that data would be of the highest quality as it is captured in clinical and operational systems for the purpose of direct patient care, and that assumption is probably true. The problem comes when you need to use this data to get a national picture of performance and for secondary uses. 

This requires data to be extracted from these clinical and operational systems in a standardised format for aggregation within a national setting. Taking mental health as an example, there are probably over 500 providers (many being relatively small) of mental health services (excluding NHS Talking Therapies); the actual number is unknown. Of those providers, we have identified at least 30 distinct IT systems in use (such as SystmOne, Epic and RiO), as well as many in-house systems. The data within these systems is held in differing structures and formats and key information will be captured as free text. 

To create the national collection for Mental Health (MHSDS – Mental Health Services Dataset) requires providers to make monthly record level submissions to NHS England. The provider must use the technical output specification (TOS) and user guidance to understand the scope and definition of each data item to be submitted. In addition, they have to familiarise themselves with the MHSDS intermediate database to understand how data items are grouped for the data submission file. To achieve this, providers have to carry out a ‘data mapping exercise’ to understand how well their existing systems align to the MHSDS TOS and take appropriate action to ensure that the standard is fully met. As Mental Health is a multifaceted service covering many policy areas, the data is complex and the submission process can be arduous, especially for smaller providers. 

Supporting Providers to Submit Data

A key focus has been on increasing the number of mental health service providers submitting to the MHSDS, which now stands at over 370 providers on a monthly basis and an estimated 99%+ of all NHS funded activity. This improvement from 85 providers in 2016 has been achieved through a variety of initiatives.  

Understanding who should be submitting is key and once they are submitting, knowing what services they provide and therefore the data they should be submitting. A Master Provider List is maintained that identifies all known providers in scope of the MHSDS together with their submission behaviours. Data submissions are tracked throughout the submission window, which together with historic submission behaviours result in tailored communications being sent to providers to encourage more positive behaviours. 

Ensuring All Data is Submitted

In collaboration with the CQC, regional leads and providers, it has been possible to identify the services that are being delivered by most providers. This has allowed us to assess whether providers are submitting all relevant data. This has been a particular problem with Independent Sector Service Providers (where they are delivering NHS funded services) and has required the intervention of DHSC and Health Ministers. 

Improving the Quality of Data

Once providers are submitting data across the service lines that they provide, we can assess the quality of that data and support providers to improve it. This starts with self service tools, at the point of submission providers receive a line-by-line data quality assessment. This can be a daunting report, hence a Validation and Rejection Submission Tool was developed that converts the record level submission report into an easy to understand summary of the issues, with instructions on how to fix them. 

Providers can resubmit the data as many times as they want within the submission window, improving the data each time. However, there are occasions where data issues are identified after the submission window is closed that can affect the quality of the data for that month but also have a knock-on effect on future monitoring, such as for 3 or 12 month rolling metrics. To address this issue a multiple submission window model (MSWM) was implemented to allow providers to address data quality issues throughout the financial year. Use of the MSWM is closely monitored and reported upon to avoid abuse of the facility, as it should be a last resort. 

To illustrate the quality of the data and the compliance of that data it is surfaced within a data quality dashboard that reports on the quality of each data item submitted by each provider, allowing for comparisons at a provider, regional and system supplier level. In addition, to promote the compliant use of SNOMED (a structured clinical vocabulary for use in electronic health records) relevant data items are reported upon within a SNOMED dashboard. The dashboard assesses how much SNOMED is flowing to MHSDS, to which data items and tables, and by which providers, there is also a focus on correctly identifying procedures and assessments. 

To reinforce these tools, providers receive an automated data quality report by e-mail each month. These reports summarise the key issues with a provider’s data and strategies to fix them. Providers can also access policy specific guidance in the form of workshops, webinars and documentation. This has previously focussed on topics such as eating disorders, restrictive interventions, problem gambling, perinatal and maternal services and memory clinics. In addition, providers have received questionnaires to better understand where they need more support. 

Talking About Data Quality

While all these tools facilitate better data it is the direct engagement with providers by the data liaison team that can have the biggest impact. The data quality analysis will identify the providers experiencing the biggest challenges, which is used by the data liaison team to provide tailored support on a 1-2-1 basis. This was particularly successful during the recent Advanced Cyber Incident that impacted the data of a variety of mental health service providers for almost 9 months. 

Next Steps in Improving Mental Health Data

At first sight all these solutions may seem excessive. However, the data is the foundation for decisions relating to commissioning, service improvement and service design, it supports research and innovation, and helps understand the impact of mental health care on patient outcomes and experiences. Through improvements in data quality, we have been able to close several duplicate collections and are now able to move to a single window for data submission. This will soon allow insights to be delivered a whole month earlier, making decisions more relevant and timelier. 

At the start of this blog, I stated that the data we are referring to is secondary uses data, but that data did originally come from clinical and operational systems that are used for direct patient care. As we know that this data is of a higher quality than that within MHSDS, we must find a way to mitigate the degradation in quality that we are currently seeing. Initiatives of this nature are currently being explored within NHS England in the hope that we can improve data quality further, make the data even timelier, as well as easier to collect. 

The public good of statistics – narratives from around the world

In our latest guest blog, Ken Roy, independent researcher and former Head of Profession, reflects on the narratives from international statistical systems, linking them to the public good that statistics can serve…

I really welcome the decision of the Office for Statistics Regulation (OSR) to pursue research and engagement work on the ‘public good’ of statistics. It feels sensible and timely to explore this concept that is (legislatively) at the heart of the UK’s Official Statistics system.

Last year’s report by the OSR in collaboration with Administrative Data Research UK (ADR UK) captured views on this topic from members of the public – exploring the potential public good arising from the use of administrative data in research. This provided some really helpful insights – with participants framing ‘public good’ both in terms of processes (e.g. appropriate data protection) and of tangible outcomes (for people and for communities).

That focus on tangible outcomes links to my fascination about the narratives used to describe the positive impacts of statistics, specifically of Official Statistics – linking to my research interest in the future choices that Official Statistics systems might need to make (including about what and who they are for).

Narratives from official statistics systems

I have been looking at some of the narratives used by bodies producing Official Statistics – specifically those in a sample of recent strategies and business plans from different National Statistical Offices. Inevitably these documents focus on planned programmes of work – the key statistical outputs, the technical and methodological investments etc – and occasionally on interesting things like budgets.

When these documents touch on the rationale for (or purpose of) Official Statistics, one approach is to present Official Statistics as a ‘right’ of citizens or as essential national infrastructure. For example Statistics Finland frame Official Statistics as “our shared national capital”. A further common approach is to reference the broad purpose of improved decision making – Statistics Canada has the aim that “Canadians have the key information they need to make evidence-based decisions.”

Looking beyond these high-level statements, I was keen to find any further, more specific, expressions of real-world impacts. The following sets out some initial groups of ideas and some representative quotes.

In terms of direct impacts for citizens, some strategies have a headline aim that citizens are knowledgeable about their world – Statistics Iceland aims to enable an “informed society”. A slightly different ambition is that different groups of citizens are represented or ‘seen’ by Official Statistics. The UK Statistics Authority aims to “reflect the experiences of everyone in our society so that everyone counts, and is counted, and no one is forgotten”. There are also references to the role of Official Statistics (and data more broadly) in empowering citizens – most commonly through giving them the means to hold government to account. One of the headline aims of New Zealand’s Data Investment Plan is that “government is held to account through a robust and transparent data system”.

Also relevant to citizens is the ambition for Official Statistics to enable healthy, informed public debate – one aim of the Australian Bureau of Statistics is that their work will “provide reliable information on a range of matters critical to public debate”.

Some narratives hint at the contribution of Official Statistics systems to national economic success. Stats NZ notes that “the integrity of official data can have wide-ranging implications … such as the interest charged on government borrowing.” The Papua New Guinea statistics office references a focus on “private sector investors who want to use data and statistics to aid investment decisions”.

Finally, we come to governments. Official Statistics are regularly presented as essential to a better, more effective, government process – through establishing understanding of the circumstances and needs of citizens, businesses and places and hence supporting the development and implementation of better policies, programmes and services in response. The National Bureau of Statistics (Tanzania) sees Official Statistics as enabling “evidence-based formulation, planning, monitoring and evaluation which are key in the realization of development aspirations.” A related theme is the contribution to good governancethe United Nations presents Official Statistics as “an essential element of the accountability of governments and public bodies to the public in a democratic society.”

Some reflections

It has been illuminating and enjoyable (honest!) to scan a small sample of corporate documents for ideas about the impacts of Official Statistics, recognising (as you will find if you click any links) that this is a bit of a mining exercise. A more rigorous exercise would be able to better account for the various factors (including administrative and cultural norms, and language) that shape what is included (and what is not) in these sources.

There are clearly common themes across these documents – I have not attempted to create any ranking of key phrases but I suspect that ‘evidence-based decision-making’ might come top. Where documents do go beyond universal purposes, there are some interesting ideas to build on. For example, can we better articulate (or quantify) the public good that might come from different groups of citizens being better represented within Official Statistics or from public debate being better informed?

One challenge, however, is that when more detailed impacts are considered, we start to enter the world of priorities and trade-offs. Hence public bodies producing Official Statistics seem to have to find a balance between setting out high-level outcomes or stating more specific ambitions.

From the documents that I have looked at, no National Statistical Office has been brave enough to commit to delivering a full outcome set of knowledgeable, represented, empowered citizens, healthy public debate, economic success and effective government (demonstrating good governance).

All of which suggests that expressing the public good of Official Statistics is complicated and that there are different approaches that might be adopted or evolved. In a UK context, it is therefore encouraging that gathering further evidence on this topic this remains a priority area within the OSR’s Areas of Research Interest.

If tangible outcomes are to be part of that continuing conversation, then it might be worth trying to make some sort of connection between the sorts of narratives referenced above and the views of members of the public. It would be interesting to know whether the participants in the original OSR and ADR UK research work had any of these Strategy or Corporate Plan derived ideas in mind when they said that they wanted to see tangible impacts from the use of statistics. Perhaps we should ask them.

Keeping your statistics (and your garden) sustainable

In our latest blog, Rob Kent-Smith, Deputy Head of OSR, discusses what statistics producers should think about when reviewing their outputs…

As Autumn has arrived, the keen gardeners amongst us will be pruning and thinning out fruit bushes and perennials to make sure that our gardens remain sustainable and to promote fresh growth in the seasons ahead.

There are some parallels in the world of statistical production, it is important that the range of statistics produced are sustainable, meet the needs of users and have room for growth to respond to emerging topical needs. The Code of Practice for Statistics provides a framework to do this and gives flexibility and support to make the right decisions in the right way.

Our state of the statistical system report found many good examples of producers developing new statistics and analysis to respond to the topics of the day, with recent topics including the cost of living and data on those seeking sanctuary in the UK from the war in Ukraine.

But additional work means additional effort and resources, and we are in a time where many statistics producers are facing tight budgets. To maintain this level of responsiveness it is more important than ever to review how statistics best meet user needs.

Here are five things producers should think about when reviewing their outputs;

1. The needs of users should be front and centre.

Producers regularly engage with their users; this means they know what matters to users and can make informed decisions on what statistics meet their needs. Producers are sometimes required to reprioritise and make decisions about their outputs at pace, so it is important to ensure evidence and documentation on user needs are captured and continually updated. For some decisions, this may be sufficient. However, we know the range of users some producers engage with could be wider so producers should supplement their existing intelligence with metrics they have on usage and impact. They should also consider whether additional engagement and consultation is appropriate, particularly when proposed changes are significant or there is a wider user base whose needs are not fully understood. Involving users in the process can help secure buy-in and support. It also enables users to understand the pressures facing producers to meet wide ranging needs, assist with making hard choices and identify options to mitigate any data gaps that may result. Sometimes, it is not always possible to meet the needs of all users, in these cases it is important to explain which users’ needs are being prioritised and what needs cannot be met and why. Our guidance on user engagement will be helpful to support you.

2. The same principles should apply to different types of statistics.

OSR introduced the term ‘Accredited Official Statistics’ to describe National Statistics in September 2023 – you can find more about this. Some producers have told us that they interpret accredited official statistics as being more important than official statistics or official statistics in development, and that accredited statistics should be prioritised when considering changes to outputs. However, decisions on prioritising statistics should be based on maximising the value of the statistics to users and in turn the public good, regardless of their badging. It might be that a new statistic in development could be a higher priority and more valuable to users than a long-established accredited official statistic.

3. Increasing efficiency or reducing content can be a good alternative to stopping outputs.

The Government Analysis Function Reproducible Analytical Pipeline (RAP) strategy, outlines an approach to improve efficiency in the production and quality assurance of statistics and analysis, alongside other benefits. Alternatively, reducing the frequency, detail, or supporting material of a statistical release can enable savings while better meeting user needs, this could be a good alternative to stopping a statistical release. In making these decisions ensuring the continued quality of the output is fundamental.

4. Communication of any changes should be clear.

It is important that decisions on the future production of statistics are clearly communicated to users in advance. These communications should set out the changes users can expect, the rationale for those changes, how the decisions have been arrived at and how users can engage with producers to feedback on the approach. Producers should include information on relevant statistical releases and inform known users through existing communication channels. For larger or more significant changes producers should consider a public statement that sets out their plans.

5. Get in touch with us.

Regularly reviewing your range of outputs to ensure they best meet user needs is an important part of ensuring the public good and we are here to provide guidance and support on how these changes can be best made following the Code of Practice for Statistics.

In line with our State of the Statistical System report, we will be doing more work in this area, aiming to improve the accessibility of our work and guidance on reducing outputs. But in the meantime, don’t forget to prune the garden!

The sociotechnical nature of data sharing and linkage

In our latest blog, Head of Development and Impact Helen Miller-Bakewell, and Director General for Regulation Ed Humpherson discuss the state and sociotechnical elements of data sharing and linkage across government… 

We published our report on data sharing and linkage in July. The report highlighted that data sharing and linkage in government is at a crossroads. Not all the routes from the crossroads lead to positive outcomes. The report set out 16 recommendations to ensure that the positive path – data sharing for the public good – is that one that is taken. 

In highlighting this position, we were not taking a purely technical approach. We were not looking just at the technical and legal underpinnings of data sharing and linkage and assessing progress against them. We were not just looking at data standards and architecture. 

For sure, technical and legal issues do appear in the report, and getting them right is a key enabler. 

But the crossroads analogy comes from the broader perspective the report brings to data sharing and linkage. In the report, we consider four possible ‘future scenarios’ for data sharing and linkage, set five years from now. They are not predictions but stylised versions of possible futures, which help to bring out the impact on public good of acting on (or not acting on) the current barriers that exist to data sharing and linkage. 

Because we wanted to look at data sharing and linkage not just in a technical way but in its broader social context, each scenario explicitly considers public understanding and buy-in to the benefits of data being shared and linked. And our future scenarios feature personas – people who work with data inside and outside government – and how the different scenarios impact on them and the things they are interested in. 

In doing so, we consciously adopted a sociotechnical approach. It has helped us to demonstrate that public engagement and social licence are intrinsically linked to the ability for linked data to serve the public good. Without public support, data are less likely to fulfil their potential.  

This term – sociotechnical – first came up in our work through the Ada Lovelace Institute. When we published our report on exam algorithms, Carly Kind of the ALI tweeted that she liked OSR’s ‘sociotechnical’ approach. At that time, it was not something we incorporated into our work. 

However, being generally curious by nature, in OSR we started to wonder what lay behind the idea of behind sociotechnical. 

We learnt that the sociotechnical denotes an interest in how technology, and technical issues more broadly, both influence and then in turn are influenced by how people behave, respond and think about them. I picture it as a sort of dance between the social – the human and community aspect of something – and the technical – which is the abstraction of something into a measurement or calculation structure of some kind. 

The more we looked into this, the more we realise that we have always worked in a sociotechnical way. We care about how people use statistics – that is what confers value. We care about the trustworthiness of the organisational processes, which is about human issues like the role of the head of profession. And we care about quality, not in an abstract computational sense, but whether the statistics provide users with a good estimate of what they are aiming to measure. 

For the data sharing and linkage work, we established a sociotechnical panel of advisors – people who we’ve met or worked with who have a good handle in this interface of the technical and the social. They were They were Nick Pearce, Brian Rappert, Rachel Coldicutt, Jessica Davies and Sabina Leonelli. 

The panel was invaluable. They challenged the way we thought about public engagement. Our early thinking was too fixed. We talked of “engaging with the public”, as if there was a single public and a single best way to track public approval. The panel reminded us that the public is not homogenous and counselled us to advocate for public engagement that is targeted, based on the intended outcomes of projects and the demographics of the population that are most likely to be affected. And they helped expand our thinking on the connection between public willingness to share data and their trust in different types of organisation. The more subtle focus on the ever-evolving, context-dependent social licence for data sharing in the report came from the advice of this panel. 

Finally, it is not just social context beyond government that can impact data sharing and linkage. Culture and people within government can be key determinants of progress. During our review we heard that, at every step of the pathway to share and link data, the people involved are instrumental to determining whether projects succeed or fail. We heard examples of departmental barriers becoming unblocked when new people arrive showing how many can be overcome simply with a new motivation, knowledge or skill. 

Reflecting on the report, we set out to write about data sharing and linkage – a potentially dry and technical topic – and it ended up being more about people than technical architecture. And, in fact, we find this is a common pattern across almost all our work.  

The final report, then, is an important diagnosis of the state data sharing and linkage in government. It is also the fullest expression yet of the sociotechnical angle to our work. 

 

Related Links:

Data Sharing and Linkage for the Public Good

Shining the spotlight on the quality of economic statistics

In our latest blog, Statistics Regulator Emily Carless discusses the importance of quality in economic statistics…

It’s an exciting time to be a regulator of economic statistics in the UK. The economic statistics landscape is changing, with more new and innovative data available than ever before. The challenges presented by increases in the cost of living have also put greater attention on economic data.  The regulatory landscape has also changed: the UK’s departure from the EU means the European statistical office will no longer have a role in verifying the quality of UK statistics. This has created an opportunity for us, as the independent regulator of official statistics in the UK, to develop a new programme to provide continued assurance to users on the quality of economic statistics.

Since March this year I have been leading this programme, which we are calling the ‘Spotlight on Quality: Assuring Confidence in Economic Statistics’.  As part of this programme, we are carrying out a series of quality-focused assessments. To help us do this, we are developing a quality assessment framework. We started with the Quality pillar of our Code of Practice for Statistics, and have adapted it by taking relevant practices from other international frameworks to enable a deeper dive on the many elements of quality in economic statistics.

Earlier this summer we published the report of the first pilot assessment on Producer Price Inflation statistics, which are produced by the Office for National Statistics (ONS). We gathered feedback on the assessment and the programme more generally to get insight into how we can maximise the benefits of the programme. My colleague Job led the first pilot assessment and has this to say:

“ONS’s Producer Price Indices (PPIs) were an excellent candidate for the first quality-focused assessment. The new quality assessment framework that we developed helped us investigate all aspects of quality in greater detail than we normally would in an assessment. We’re continuing to refine the framework to make it an even more effective tool. One thing we trialled with this assessment, which worked particularly well, is speaking to the international statistical community about the UK’s statistics. Conversations with price statistics experts in other National Statistical Institutes gave us an insight into the extent to which ONS is following international best practice in producing PPIs. We’ll look to replicate this approach in other quality-focused assessments.”

We received positive feedback about our quality focused assessments from the experts we spoke to in other NSIs. Espen Kristiansen of Statistics Norway commented on the benefits to the international community of our programme:

“It is very valuable to have an external review of statistics. It reminds us how we can frame a discussion about quality, which for statistics is a surprisingly complex and many-faceted concept. Not only can the review detect flaws, it also makes us better at continuously improving our statistics in the future.”

Users of economic statistics in the UK have also found our quality-focused assessment on the PPIs useful. Cheryl Blake from the Ministry of Defence said:

“We welcome the findings of the OSR report on the quality of the PPIs and the opportunity to provide user feedback, particularly the recommendations to publish weights and standard errors which will greatly help our team better understand the underlying drivers of the indices and their quality. This is of growing importance as there is growing interest across Government in using industry-specific PPIs for indexing contracts to better track prices and provide a fairer return to industry. Further information on sample size and resultant quality will help inform index selection and we hope to work with ONS regarding the rationalising on index publication to ensure the provision of critical indices for indexing Government contracts.”

In carrying out these assessments we aim to be supportive of the producers of economic statistics, to champion where they have improved the quality of their statistics as well as to challenge where further improvement is need. Chris Jenkins, Assistant Deputy Director for Prices division in ONS, highlighted the benefits of these assessments to the producer team:

“The regular churn of our monthly statistics sometimes means we don’t get the opportunity to review the methods and metadata that supports our production process as frequently as we would like. This assessment provided us with the perfect opportunity to take stock of what we do, and the constructive support from OSR highlighted areas where we are doing really well, but also areas where we need to make quality improvements. Having the assessment now gives us a clear set of actions we can take to improve the quality of PPI, which is a key economic indicator.”

The outcomes from our quality-focused assessments also provide useful information on where the quality of other statistics could be improved. We will work with statistics producers and the Statistical Heads of Professions to share good practice and shine a light on where the quality of statistics more broadly can be improved. Rachel Skentelbery, ONS Deputy Head of Profession for Statistics, said:

“We welcome OSR’s new quality-focused assessment programme which complements work we are leading to support ONS colleagues in assessing and improving the quality of statistics. Many of the findings identified in these assessments will be relevant beyond the specific output being reviewed, for example in areas such as RAP [reproducible analytical pipelines], quality assurance, user needs and transparency, and we look forward to working together to share lessons and promote best practice to drive improvements to quality across the organisation.”

Over the next few months we will be publishing the report of our second pilot quality-focused assessment on the Profitability of UK companies and Gross Operating Surplus of private non-financial corporations, and publishing the quality assessment framework that we have developed for use in these assessments.

 

If you are interested in learning more about this programme or have any feedback on our first report then please get in touch by emailing regulation@statistics.gov.uk.

How do people use official statistics to make decisions?

Sofi Nickson, Head of Research at OSR, shares why OSR is interested in the role statistics play in decision making by members of the public along with what we know about this so far, and invites others to share evidence they have on the topic.

When I first heard about the Office for Statistics Regulation (OSR), I assumed it simply checked whether statistics are accurate or not. It wasn’t until last year, when I happened upon a job advert for OSR that I looked into what it really does. It turned out that I was somewhat off the mark in my assumption – OSR’s vision is not as it happens limited to ‘accurate statistics’, but it is the far more inspiring ‘statistics that serve the public good’. For the past few years, colleagues across OSR have deepened their understanding of what this may look like through their regulatory work and supporting research programme, which I am now lucky enough to lead. Part of the research programme is understanding the role official statistics can play in decision making by members of the public and, in this blog post, I explain why we are interested in this and what we know so far, then I invite you to share your thoughts on the topic.

Statistics serving the public good

The Statistics and Registration Service Act states that serving the public good includes assisting in development and evaluation of public policy, and informing the public about social and economic matters. ‘The public’ here could be anyone outside of government, in fact a report from workshops on whether scientists understand the public states that ‘there’s a thousand publics out there that one could address, any of whom has to be understood… in order to know how to deal with them, how to work with them, engage them, try to benefit them’. We have begun understanding how some publics play a role in statistics serving the public good, such as non-governmental organisations that may use statistics to provide services, businesses who can use them to adapt their practices and better meet needs, and we have evidence from analysing applications to access public data suggesting that academics see themselves holding a role in providing an evidence base for decision making. Even the media plays a part, with ESCoE research finding that journalists help translate statistics for public consumption.

The point here is that for statistics to serve the public good, they must be a tool both for government and for those beyond. When we look at statistics use outside of government, we have heard from a wide range of civil society organisations through our regulatory activities about their uses of statistics on behalf of the public. This is a way for statistics to serve the wider public indirectly, without individuals using statistics themselves. However, we are currently missing an important piece of the puzzle – what use looks like for individual citizens outside of such organisations. This family or individual-level use of statistics is far less visible, but may be no less valuable in serving the public good. In OSR we want to shine light upon these hidden uses of statistics by exploring how individual citizens use statistics in their professional and personal lives, and what they value in statistics.

The more we can understand, the better we can ensure our regulatory decisions and recommendations support members of the public statistics users. This topic is vast, so to narrow it down we are starting with how individual citizens use official statistics to make decisions, and we are focussing on three areas:

  1. Whether members of the public find value in using statistics to make decisions (including whether they do use statistics at all)
  2. Whether members of the public feel equipped to use statistics to make decisions in the way they are currently presented
  3. How statistics inform decisions (including how much influence they have alongside other factors).

Do members of the public find value in using statistics to make decisions (and do they use statistics at all?)

A quick search about using statistics to make decisions gives lots of potential examples. For example, using statistics on school performance to inform where you want to educate your child, or using statistics on crime to decide where to live. But we currently lack evidence about whether and how these potential uses play out in real life – do people actually use statistics like this, or do we just think they could. Even more specifically, do people use official statistics, or are their information needs being met by other sources.

In the Public Confidence in Official Statistics 2021 survey, just over half of respondents (52%) agreed to some degree that statistics helped them to make a decision about their lives. However, we don’t know from this what sort of statistics or decisions respondents were thinking about. We also don’t know what might support the other 48% of respondents get value from using statistics as well, or even whether this 48% want to use statistics at all. It may be that individuals are already satisfied with organisations in civil society using statistics on their behalf.

Do members of the public feel equipped to use statistics to make decisions in the way they are currently presented?

From our commissioned research into statistical literacy we saw great variability amongst the general public in skills linked to statistical literacy, and have concluded that responding to this is all about the communication. We have evidence on how to communicate statistics to non-specialists, for example recommendations from a programme of work by ESCoE which explores communicating economic statistics to the public. Despite strong recommendations, there is more that could be done to improve the communication of statistics to non-expert users, which is why in our business plan for 2023/24 we commit to championing the effective communication of statistics to support society’s key information needs. We don’t profess to know everything in this area though and are always interested in learning more.

How do statistics inform decisions (and how much influence do they have among other factors)?

We have uncovered an abundance of literature about human decision making, including how heuristics and biases sit alongside ‘rational’ evidence-based choices. From this we recognise that it is unlikely anyone bases their decisions on statistics alone, but we still don’t know how influential official statistics are and where they sit alongside other evidence. Are they seen as compelling and trustworthy? What factors influence this?

Can you help?

If you have read this far it will be clear that we have a lot of questions about how statistics can serve the public good. In OSR, asking questions is in our nature – as a regulator our judgements and decisions are informed by the evidence we have, so we are always seeking to learn more. If you know of any research, examples, or information that you think could inform our understanding of the role official statistics play in how members of the public make decisions then we would love to hear from you – please get in touch with us at research.function@statistics.gov.uk