Office for Statistics Regulation

How OSR secures change

Why do organisations do the things they do? Strip away all the language of business plans and objectives and strategies, and what it often boils down to is wanting to achieve some kind of positive impact.

It’s important to remember that as we launch our latest business plan for 2022/23. In this blog, rather than highlight specific outputs and priorities, I want to talk more generally about how OSR, as a regulator, secures positive change.

By change, we mean regulatory work or interventions that ensure or enhance statistics serving the public good. There are basically two ways in which our work leads to statistics serving the public good. Our work can:

secure a positive change in the way statistics are produced and/or presented; and/or
make a direct contribution to public confidence and understanding.

OSR clearly does secure impact. In response to our reports, producers of statistics make changes and improvements to their statistics and other data. Statistics producers also use the presence of OSR in internal debates as a way of arguing for (or against) changes – so that OSR casts a protective shadow around analytical work. OSR can also secure changes to how Ministers and others present data. And OSR also achieves impact through getting departments to publish previously unavailable data. In all these ways, then, OSR secures impact, in the sense of statistics serving the public good to a greater extent.

In terms of formal statutory powers, the main lever is the statutory power to confer the National Statistics designation. This in effect is a way of signalling regulatory judgement. The regulatory role is to assess, i.e. to review and form a judgement and on the basis of the judgement, that National Statistics designation is awarded. We have recently been reviewing the National Statistics designation itself.

A further power in the Statistics and Registration Service Act is the power to report our opinion. Under section 8 of the Act, we are expected to monitor the production and publication of official statistics and report any concerns about the quality of any official statistics, good practice in relation to any official statistics, or the comprehensiveness of any official statistics.

These statutory powers do not create influence and drive change by themselves. We need to be effective in how we wield them. We have to supplement them with a powerful vision, good judgement, and effective communication.

The power of ideas

The most significant source of influence and impact is the power of the ideas that underpin the Code of Practice for Statistics. The Code is built on the idea that statistics should command public confidence. It is not enough for them to be good numbers: collected well, appropriately calculated. They must have attributes of trustworthiness, quality and value.

The power of these ideas comes from two sources. First, they are coherent, and in the Code of Practice, are broken down into a series of increasingly granular components – so the ideas are easy for producers to engage with and implement. Second, they have enormous normative power – in other words, trustworthiness, quality and value represent norms that both statisticians and, senior staff want to be seen to adhere to, and wider users want to see upheld.

These powerful, compelling ideas represent, then, something that people want to buy into and participate in. A huge amount of OSR’s impact happens when OSR is not even directly involved – by the day-to-day work of statisticians seeking to live up to these ideas and the vision of public good that they embody.

Judgements

OSR’s work begins with the ideas embodied in the Code, which we advocate energetically, including through our crucial policy and standards function. The core work for OSR actually consists of making judgements about trustworthiness, quality and value, in multiple ways:

our assessments of individual sets of statistics, where we review statistics, form judgements, and present those judgements and associated requirements to producers and then publicly – either through in-depth assessment reports, or by quicker-turnaround reviews of compliance;
our systemic reviews, which address issues which cut across broader groups of statistics, and which often focus on how of we statistics provide public value, including highlighting gaps in meeting user needs;
our casework, where we make judgements about the role statistics play in public debate – whether there are issues with how they are used, or how they have been produced, which impact on public debate; and
through our broader influencing work, including our policy work, and our research and insight work streams.

These judgements are crucial. Our ability to move fluidly using different combinations of our regulatory tools is important to securing impact. It allows us to follow up where the most material change is required and extend our pressure – and support – for change.

We are able to make judgements primarily through the capability of OSR’s people. Their capability is strong, and we depend on their insight, analysis, judgement and ability to manage a range of external relationships.

Communication and reach

It is not enough for us to make good judgements. We need to make sure that any actions are implemented – in effect, that our judgements radiate out into the world and lead to change.

There are three main channels for achieving this reach:

Relationships with producers: our relations with producers are crucial. Heads of Profession and the lead statisticians on individual outputs are the key people for making improvements; their buy-in is crucial.
Voice and visibility: having a public voice magnifies our impact. It ensures that policymakers are aware of what we do and understand that our interventions can generate media impact.
Wider partnerships: while our direct relationships with producers, and our public voice, can also create sufficient leverage for change, we also draw on wider partnerships. For example, credible external bodies like the RSS and Full Fact can endorse and promote our messages – so that producers face a coalition of actors, including OSR, that are pushing for change.

And we put a lot of emphasis on the views, experiences and perspectives of users of statistics. Almost all our work involves engaging with users, finding out what they think, and seeking to ensure producers focus on their needs.

In that spirit, we’d be very keen to get reactions on our own business plan – from all types of users of statistics, and also from statistics producers.

Conclusion

Business plans should not simply be a list of tasks. It is also important to be clear on how an organisation delivers, how individual projects and priorities help achieve a positive impact. In OSR’s case, achieving this impact involves the power of ideas, good judgement and effective reach.

With this clarity around impact, our business plan (and work programme) comes to life: more than just a set of projects, it’s a statement of ambition, a statement of change.

But the business plan is also not set in stone. We are flexible and willing to adapt to emerging issues. So if there are other areas where we should focus, or other ways we can make a positive difference, we’d really welcome your feedback.

Guest Blog: National Statistics – The Road to Accreditation

The Office for Statistics Regulation has today designated the Family Practitioner Services statistics in Northern Ireland as National Statistics.

In this guest blog, Martin Mayock, Head of the Family Practitioner Services (FPS) Information Unit in Northern Ireland’s Health and Social Care (HSC) Business Services Organisation (BSO), and a Senior Statistician in the Northern Ireland Statistics and Research Agency (NISRA), discusses his experience of the assessment process – which wasn’t, in fact, as daunting as his team first thought!

By way of background, Information Unit had always produced a wealth of information across primary care: medical, ophthalmic, dental and pharmaceutical services for internal use by our health policy and operational colleagues. We even, on occasion, publicly released various ad-hoc reports and tables but were a long way short of being a Code-compliant organisation. This despite BSO being a legally specified producer of Official Statistics (OS) since 2012. It’s not that we didn’t recognise the importance of compliance, it was simply a matter of resources and carving out sufficient time from our day-to-day analytical project support, to progress our OS aspirations. The key was good planning and accepting that this would not be done overnight, taking whatever time was necessary to update our processes, plug identified gaps and develop documentation. Of course, we needed the backing of senior management if we were going to be “distracted” from the day job so we first had to sell the benefits.

Our first milestone was to release an Annual Compendium in 2018, covering all of our key primary care areas, complying with the Code as far as possible. A quarterly series soon followed and by our third year, responding to user demand, we were ready to split the compendium into separate service areas with a dedicated team responsible for each. User engagement was a key component of the work programme with readership surveys, supplemented by targeted stakeholder interviews, allowing each release to evolve in a way most beneficial to its user base. The fact that our teams were both users, through our ongoing project work, and producers of the data helped enormously in improving its quality and offering guidance on its use.

By our fourth year, and following two successive releases of our service specific publications, we were finally ready to push the button and subject our outputs to the all seeing eye of the OSR. Yes, it had taken a few years to get to this point, albeit from a fairly low base, but we now felt confident that our processes were in order and we had a good story to tell. The invite was duly issued in November ‘21 and we had our assessment initiation meeting with the OSR team, headed by Dr Anna Price, in January this year. Everything was clearly explained to us in terms of how the process would run and what would be expected from us by way of evidence. The OSR team, comprising Anna, Jo Mulligan and Sarah Whitehead, were really open and friendly (surely a ruse to get us to lower our guard lol) and keen to help with various initiatives that we were planning such as the introduction of Reproducible Analytical Pipelines into our production process. It all seemed reasonably straightforward and, certainly as a veteran of 4 previous assessment campaigns in other NI Departments, much less formal and bureaucratic than I had remembered – can’t last, I thought!

Roll forward to February, and we had our follow up meeting with the assessment team to discuss our submitted evidence but also, importantly, draw upon information the team had gleaned from our users. This meeting involved all of our publication leads so, with virtual flak jackets donned, we braced ourselves for the inevitable onslaught. But, again, we were pleasantly surprised. The meeting was more like an interesting chat around our various processes, with helpful suggestions and resources offered which could further enhance our outputs. Of course, there were queries and clarifications sought, some of which were followed up in writing in the weeks that followed, but these were conveyed in a constructive way and the different perspective offered us an opportunity to highlight aspects of our process that we’d overlooked in our initial evidence submission.

We received first sight of our draft assessment report the following month and I admit to opening the document with a feeling of slight trepidation. I’d had the impression that the team felt we were in reasonable shape from our meetings but there’s always requirements, I mean they have to find something, right? But no, I read the report twice, definitely no requirements! What it did contain was a succinct summary of how we matched up against the Code of Practice pillars of Trustworthiness, Quality and Value along with some helpful suggestions of where we could enhance our outputs against these. Several useful resources which we might find useful in this regard were also signposted. We had an opportunity to suggest any amendments and correct any factual inaccuracies, of which there were very few, and several weeks later we were notified that the report and recommendation to designate our outputs as National Statistics had been accepted by OSR Regulation Committee. Our journey was finally complete! Of course, it never really ends and we will need to continue to improve and innovate to ensure standards are maintained and the needs of ever more demanding users continue to be satisfied.

Before signing off, I thought I’d leave you with what I feel were the three most important factors in helping us achieve our designation so painlessly.

Invest time achieving proper buy-in from senior management within your organisation – you will need their support to allow you to spend time developing aspects of your processes that they may not immediately see as being important to their core business.
Prepare, Prepare, Prepare – Don’t rush to get your outputs assessed, wait until you are properly ready. We were also able to draw upon the support of our colleagues in the Northern Ireland Statistics and Research Agency (NISRA) who provided lots of good advice and resources, including other relevant assessment reports. The assessment focus can change over time and statisticians are constantly innovating. We can learn a lot from our peers. I’m not saying take 4 years but, if you invite an assessment too early, then you will leave yourself with a limited window, typically 3 months, in which to meet requirements. This could feel like a burden on top of your business as usual. Better to meet as many potential requirements ahead of time as possible.
Have similar outputs assessed as a batch – it might seem tempting to submit individual outputs for assessment in order to make the process more manageable or you may feel that some are more ready than others. However, there can be synergies between outputs and processes that make sense to consider together. We also included all of our publication leads when we met with the assessment team and this all helped deliver a more rounded and efficient assessment.

In my experience, the assessment process itself has definitely evolved for the better and feels more like a collaborative venture these days rather than a statistical audit. It definitely feels more light touch than previously and, although a lot of hard yards are still required to ready your outputs, it is great to see that your efforts will be recognised by the assessment team.

The Family Practitioner Services statistics in Northern Ireland assessment report has been published today, less than six months since the initial invite for assessment.

All-in-all we found it to be a very worthwhile and positive experience so if you are thinking of taking the plunge then go for it, you might just be pleasantly surprised!

Time doesn’t stand still

In light of our first change to the Code of Practice, which relates to the release times of official statistics, and gives producers the opportunity to release statistics at times other 9.30am, where they and we judge that this clearly serves the public good, our latest blog sees Penny Babb – Head of Policy and Standards – look back over the first four years of the Code of Practice and consider its impact.

In some ways, time seems to have stood still over the last couple of years. But in other ways, I’m amazed it is only four years since we published Edition 2.0 of the Code of Practice for Statistics – how time has flown!

As I contemplate what the impact of the Code has been since we launched it in February 2018, my abiding sense is that the heart of the Code has never been better understood or more comprehensively applied. I know I’m biased, much like a proud parent sharing the latest pics of their child’s exploits, as I was closely involved in writing the Code and am OSR’s lead on its guidance.

I’m not sure there could have been a more profound test of the Code than statisticians needing to turn their work on its head and seemingly perform miracles – bringing new meaning to ‘vital statistics’. How easy it would have been to say in March 2020 that the ‘rule book’ needed to be ripped up as the world had changed.

Instead, analysts in the UK had a firm guide that supported and enabled them to make new and challenging choices – what to stop, what to change. How to better speak, explain, reach out to the massive audience with an insatiable appetite for numbers, to make sense of the unknown.

Underpinning their decisions were Trustworthiness, Quality and Value – the framework of our Code. For each dilemma faced, the answer lies in thinking TQV:

What should I release – what information is needed right now to support decision making?
How should I release it – how can you ensure that the public retains confidence in the statistics and the people releasing them?
But there’s no time for ‘full’ QA – what information do you have confidence in that will not mislead?

Ultimately the Code of Practice is not a rule book but a guidebook.

Very little in the Code is a directive. One exception has been the practice setting out the time of release of official statistics as 9.30am (T3.6). Today we are changing this practice following our three-month consultation in autumn 2021 and conversations with producers, users and other stakeholders. With a focus on what is prime in enabling the statistics to best serve the public good, producers can now consider whether there is a need to release their official statistics at a time other than 9.30am.

There is nothing special about this specific time of day. The main characteristic is that it is a standard time across all official statistics producers in the UK that helps ensure consistency and transparency in the release arrangements and protects the independence of statistics. This is an essential hallmark of official statistics, and a norm that we continue to expect and promote.

However, as the pandemic clearly showed, there are some situations when a producer may sensibly wish to depart from the standard time. The new practice enables greater flexibility, but a release time should still depend on what best serves the public good.

We will continue to protect against political interference and ensure the public can have confidence in the official statistics. The final decision on whether an alternative release time is used will be made by the Director General for Regulation – the head of the Office for Statistics Regulation. All applications will be listed in our new table – it will include rejections as well as cases where the application was approved.

And there may be further changes to the Code of Practice to come this year. For example, our review of the National Statistics designation has already proposed changing the name of experimental statistics. So, watch this space for further developments. And if you have any reflections on the Code of Practice, please feel free to send them to regulation@statistics.gov.uk.

Exploring the value of statistics for the public

In OSR (Office for Statistics Regulation), we have a vision that statistics should serve the public good. This vision cannot be achieved without understanding how the public view and use statistics, and how they feel about the organisations that produce them. One piece of evidence that helps us know whether our vision is being fulfilled is the Public Confidence in Official Statistics (PCOS) survey.

The PCOS survey, which is conducted independently on behalf of the UK Statistics Authority, is designed to capture public attitudes towards official statistics. It explores trust in official statistics in Britain, including how these statistics are produced and used, and it offers useful insights into whether the public value official statistics.

When assessing the value of statistics in OSR, two of the key factors we consider are relevance to users and accessibility. The findings from PCOS 2021, which have been published this week, give much cause for celebration on these measures, while also raising important questions to explore further in our Research Programme on the Public Good of Statistics.

Do official statistics offer relevant insights?

PCOS 2021 shows that more people are using statistics from ONS (Office for National Statistics) now compared to the last publication (PCOS 2018). In the 2018 publication of the PCOS, 24% of respondents able to express a view said they had used ONS statistics, but this has now increased to 36%. This increase may be due to more people directly accessing statistics to answer questions they have about the COVID-19 pandemic. In our Research Programme, we are interested in knowing more about this pattern of results and also understanding why most people are not directly accessing ONS statistics.

Are official statistics accessible?

PCOS 2021 asked respondents if they think official statistics are easy to find, and if they think official statistics are easy to understand. These questions were designed to capture how accessible official statistics are perceived to be by members of the public. Most respondents able to express a view (64%) agreed they are easy to find. This is an important finding because statistics should be equally available to all, without barriers to access. Most respondents able to express a view (67%) also agreed that statistics were easy to understand, suggesting that two thirds of respondents feel they can understand the statistics they want to.

However, respondents who were aged 65 or older were least likely to agree with these two statements. Statistics serving the public good means the widest possible usage of statistics, so this is an important finding to explore further to ensure that older respondents are able to engage with statistics they are interested in. In our Research Programme, we will work to identify what barriers might be causing this effect and whether there are other groups who feel the same way too.

The value of statistics

Considering how the value of statistics can be upheld, respondents in PCOS 2021 were asked to what extent they agree with the statement “it is important for there to be a body such as the UK Statistics Authority to speak out against the misuse of statistics”. The majority (96%) of respondents able to express a view agreed with this statement, with a similar number (94%) agreeing that it is important to have a body who can ensure that official statistics are produced free from political interference. While we are cautious about putting too much weight on these two questions in the survey, these findings may at the very least indicate the public value the independent production of statistics, as well as challenges to the misuse of statistics.

In conclusion, PCOS 2021 suggests that statistics are relevant and accessible to many members of the public, but there are still some who do not access statistics or consider them easy to find or easy to understand. While the findings of PCOS 2021 offer a wealth of important information, and demonstrate the value of official statistics, it is clear there are still a lot of questions to explore in our Research Programme. We will continue our work to understand what statistics serving the public good means in practice, guided by knowledge from PCOS 2021.

Thinking about Trustworthiness, Quality, and Value across Housing and Planning statistics

When I wrote my last blog just over two years ago, reflecting on the progress that had been made on improving the public value of housing and planning statistics in the UK, little did I know that we were about to enter a pandemic. Cue homeworking, home schooling and everything else that went with it.

In that original blog, I highlighted some of the areas that we would like to see addressed and some recommendations that still needed some work such as “filling gaps on UK private rental sector levels and land use; the need for cross-government work to improve coherence on housing conditions and the quality of UK house-building statistics; and further development of UK homelessness and rough sleeping statistics.”

During those two years and against a backdrop of disruption, statistics producers adapted to the challenges of homeworking and resources being redirected to support covid-related analysis. For example, the English Housing Survey, like many other face-to-face surveys, adapted to moving online and the team was also able to provide valuable insight into the effects of the pandemic on households through its household resilience survey. Producers started to collect and publish management information on the numbers of rough sleepers and those at risk of homelessness helped off the streets during lockdown. We also saw innovation in the introduction of data dashboards such as that from the Scottish Housing Regulator which included information on the impact of COVID-19 on social landlords and their tenants.

Progress has also been made outside of pandemic-related work. For example, on the theme of private rents, ONS has recently announced further information on its redevelopment of private rental prices statistics, and Government Statistical Service (GSS) statisticians collaborated on this ONS report on Homelessness in the UK.

The cross-GSS housing and planning working group published its latest work plan at the end of 2021, and it’s great to see some ambitious work planned which tie-in to our original recommendations. The work of the group also feeds well into ‘TQV’ – trustworthiness, quality, and value – the framework of our Code of Practice for Statistics.

Think Trustworthiness

We often encourage statistics producers to publicly share their work plans to demonstrate transparency and to foster wider and better engagement with users which is why we welcome this latest work plan for 2021/22 which includes summaries of progress to date. This is an achievement in itself given all the disruption to statistics producers and teams across the GSS due to the pandemic and we would like to congratulate all involved.

It’s great that the working group has made an ongoing commitment to engage with users to help shape the work programme and development plans. It is so important for producers to listen to what users have to say and to reflect on the feedback, and that user engagement needs to be planned and collaborative.

Part of this ongoing user engagement includes continued engagement with users on what they would like to see as topics for the next coherence articles – the suggested priority areas being the private rental sector and energy efficiency – gaps that we identified ourselves through speaking to users.

Think Quality

In the Homelessness and Rough sleeping landscape we are really encouraged to see the plans set out around improving the quality of homelessness and rough sleeping data across the 4 UK countries, some of which echo our findings from our recent assessment of DLUHC’s statistics on Statutory Homelessness in England. In particular we are excited to hear that Welsh Government and DLUHC are continuing to publish their rough sleeping management information, with DLUHC’s latest Rough Sleeping snapshot including commentary on the separate management information collected since the start of the pandemic.

Also, data from the Census 2021 for England and Wales which is due to be published later this year has the potential to be used to provide further insights into the homeless populations or those at risk of homelessness.

Think Value

We are hoping that through ONS’ equalities and inclusion work, improvements to the inclusion of under-represented population groups in statistics, in particular homeless groups, will go some way to fill in gaps on the ‘hidden homeless’ such as sofa surfers, that official data do not currently capture. This is a theme that came up when we spoke to users as part of our assessment of the existing Statutory Homelessness in England statistics, and a group that some homeless charities and organisations have looked to measure.

Through our regulatory work we are starting to see more statistical producers make use of Power BI to produce more accessible and interactive content. We have already publicly commended this through outputs from our regulatory work and welcome the news that DLUHC is rolling this initiative out further across more of their statistics publications.

We would like to recognise the work at ONS on Housing Affordability statistics, with the team looking to provide further valuable insight by publishing additional measures such Purchase Affordability along the lines of their existing output on Private rental affordability in England.

We are delighted to hear of GSS plans related to housing and energy – a very topical issue at the moment – starting with looking at the energy efficiency of housing at a subnational level. This is in line with the GSS subnational data strategy ambition to produce and disseminate increasingly granular statistics. This also highlights further collaborative work and sharing knowledge and priorities, involving people from across the GSS.

Looking forward

As we head into spring, looking forward to what the rest of this year will bring, I am also looking forward to the exciting developments across housing and planning statistics which build on the hard work of GSS statisticians to improve the value of housing and planning statistics across the UK. For example, since the publication of the work plan ONS has published a research output in March looking at the feasibility of using administrative data to provide information on property types in England and Wales as an alternative to census data. We also look forward to seeing the GSS publish further updates on its progress throughout the year and an updated plan of priorities for 2022/23.

Acting against the misuse of statistics is an international challenge

That was the message of the event on 14 March hosted by the UK Statistics Authority on the prevention of misuse of statistics, which formed part of a wider campaign to celebrate the 30th anniversary of the UN fundamental principles.

My fellow panellists, Dominik Rozcrut and Steven Vale, and I discussed a range of topics, from addressing statistical literacy to regulation best practice, from memorable examples of misuse, to the cultural differences that affect public trust internationally. Although we all had different experiences and approaches, it was clear that there was a common passion for the truth and statistics that serve the public good.

The event had all the merits of on-line meetings that we’ve all become familiar with: lots of people able to join from a wide range of locations and lots of opportunities for people to engage using live chat functions.

Perhaps some people find it less intimidating to type a question into an app than to raise their hand in a crowded room, because there were lots of interesting questions asked and it was clear that the issue of preventing misuse of statistics generated a lot of interest and passion from the audience as well as the panellists

But the event also brought with it a new kind of frustration to me as a speaker: there were too many questions to answer in the time available and I felt bad that we couldn’t answer all the questions that people typed in.

So, in an attempt to rectify that, I’ve decided to use this blog to address the questions that were directly for me that I didn’t answer in real time, and those which touched on the key themes that came across during the Q&A.

“Who are the best allies in preventing misuse or building statistical literacy outside of stats offices? Are there any surprising allies?”

There are obvious allies for us, like Full Fact and the Royal Statistical Society.

I also like to give a shout out to the work of Sense about Science. Their work highlights that there is a huge amount of interest in evidence, data and statistics – and that a simple model of experts versus “the public” is far too simplistic.

There are a huge range of people who engage with evidence: teachers, community groups, people who represent patients, and a lot of others. These people, who want to find out the best evidence for their community, are fantastic allies.

And I’d also pick out a surprising ally: politicians. In our experience, politicians almost always are motivated to get it right, and not to misuse statistics, and they understand we are making the interventions we are making. So perhaps they are the ally that would most surprise people who look at our work.

“How important is statistical literacy among the media and general public in helping prevent the misuse of statistics?”

I think that having a sort of critical thinking skill is important. People should feel confident in the statistics that are published, but also feel confident that they know where to find the answers to any questions they have about them.

But equally, we need statistical producers to be better in how they communicate things like uncertainty, in a way that is meaningful for the public.

So rather than putting the responsibility of understanding solely on the user, and just talking about statistical literacy, let’s also talk about producers’ understanding – or literacy if you will – about public communication.

“You have mentioned that sometimes the stats get misinterpreted because of the way they are presented – can you share some examples?”

My point here was that misinterpretation is a consequence of what producers of statistics do. One example we’ve seen frequently during the pandemic concerns data on the impact of vaccines. It’s been the case that sometimes people have picked out individual numbers produced by public health bodies and highlighted them to argue their case about vaccines. Producers need to be alive to this risk and be more willing to caveat or re-present data to avoid this kind of misinterpretation.

“What are your views on framing statistics for example 5% mortality rate vs 95% survival rate? Both are correct but could be interpreted very differently.”

I find it impossible to answer this question without context, sorry! I definitely wouldn’t say that, as an absolute rule, one is right and the other is wrong. It depends on the question the statistician is seeking to inform. I can’t be more specific than that in this instance.

However, to avoid possible misinterpretation, we always recommend that producers use simple presentation, with clear communication about what the numbers do and do not say.

“How do we balance freedom of expression with the need to prevent the abuse and misuse of statistics?”

We don’t ban or prohibit people from using statistics, so in that sense there’s no barrier to freedom of expression. But we do want to protect the appropriate interpretation of statistics – so our interventions are always focused on clarifying what the statistics do and don’t say, and asking others to recognise and respect this. It’s certainly not about constraining anyone’s ability to express themselves.

“What’s the most damaging example of misuse of statistics that you’ve come across in your career?”

Here I don’t want to give a single example but give a type of misuse which really frustrates us. It’s when single figures are used as a piece of number theatre, but the underlying dataset from which the single figure is drawn are not available, so it’s not possible for the public to get to understand what sits behind the apparently impressive number. It happens a lot, and we are running a campaign, which we call Intelligent Transparency, to address it.

“Can you give us some more insight into how you steer clear of politics, media frenzies, and personalities?”

We always seek to make our intervention about clarifying the statistics, not about the arguments or policy debates that the statistics relate to. So we step in and say, “this is what the statistics actually say” and then we step out. And we don’t tour the news studios trying to get a big name for ourselves. It’s not our job to get media attention. We want the attention to be on what the statistics actually say.

I hope these answers are helpful, and add some context to the work we do to challenge the misuse of statistics. I also hope everyone reading this is going to follow the next series of events on the UN Fundamental Principles of Official Statistics.

The next round of events is moving on from preventing misuse, to focusing on the importance of using appropriate sources for statistics. Find out more about them on the UNECE website.

Time for a change: amending the Code of Practice to allow alternative release times for official statistics

On 5 May 2022, we’ll be making our first change to our Code of Practice for Statistics since its major overhaul in 2018. We’ll be changing two practices, which relate to the release times of official statistics, giving producers the opportunity to release statistics at times other 9:30am, where they and we judge that this clearly serves the public good. These changes are not being undertaken lightly: this blog explains how we have reached our decision and what this change will mean for producers and statistics in the future.

The coronavirus pandemic underscored the importance of ensuring statistics are shared in ways that reassure users about their integrity and independence: a big part of this is ensuring equality of access and that statistics are available to everyone at the same time. Since its inception, our Code has stated that official statistics should be released to all users at 9:30am, on a weekday (practice T3.6). However, during the pandemic the Director General for Regulation granted several exemptions for release at times other than 9.30am – to key economic statistics and to some COVID-19 related statistics.

These decisions led us to review two release practices in the Orderly Release principle of the Code of Practice for Statistics. As part of this, we ran a public consultation between September and December 2021 to consider whether greater flexibility in release arrangements should be formalised within the Code of Practice for Statistics. Today we have published the findings from our consultation, which have helped inform our decision to revise these two practices.

For most official statistics, producers, supported by several other stakeholders, told us they believe a standard release time of 9.30am best serves the public good: it enables release in an open and transparent manner, gives consistency across the GSS and protects against political interference. However, producers also said that there are some situations in which having an alternative release time can be beneficial to the public good. They would like to be able to nominate an alternative time where they see evidence of the benefit. For example, this could be where users can be better reached through targeting a statistical release that can then be promoted through the media. Enabling statisticians to speak directly to journalists and the public about their statistics has been a powerful means to ensure the statistics are well used during the pandemic.

Considering producer views and our consultation findings, we are amending principle T3.6 so that an alternative release time for statistics can be requested where the head of profession (Statistics HOP) believes that it will enable the statistics to better serve the public good. To ensure that producers remain orderly and transparent about release arrangements and can maintain statistical independence, this alternative time request will still need to be approved by the Director General for Regulation. All approvals will be listed in our new alternative release times table. The alternative standard time is to be used consistently by the producer for those statistics and should be announced in advance.

We have outlined our expectations of statistics producers seeking alternative release times in our explainer of the public good test. We highlight that considering the public good means understanding the wider nature of the use of the statistics and how the public access statistics – whether direct from producers, via the media or other means – and how the release of statistics supports both debate and decision making. Essentially, it means that producers should consider the way their statistics inform and benefit society. Being open about key decisions around release arrangements is central to maintaining the confidence of users in official statistics. To this end we are asking the Office for National Statistics to engage with its users and other stakeholders, to explain its release approach for several key economic statistics that it already publishes at 7am, and how it is balancing competing demands.

As ever for OSR, a fundamental question driving this review has been, “how can we best serve the public good?” Giving producers the opportunity to decide on alternative release times, based on their own assessment of how their statistics best serve the public good, helps us meet this ambition, which is central to both our vision as a regulator and to the Government Statistical Service’s 2020-2025 strategy.

Please see our new policy on alternative release times for further information. You can also read more about the findings from the consultation. The changes to the Code will come into effect from 5 May 2022. If you have any questions, please contact us via regulation.support@statsitics.gov.uk.

QCovid® case study: Lessons in commanding public confidence in models

Methods expert, Jo Mulligan gives an insight into the lessons learned from using the QCovid® risk calculator in commanding public confidence in models

I re-joined OSR around 15 months ago in a newly created regulator role as a methods ‘expert’ (I struggle with the use of the word ‘expert’ as how could anyone be an expert in all statistical methods? – answers on a postcard please). Anyway, with my methods hat on, myself and several colleagues have been testing the lessons that came out of our review of the statistical models designed to award grades in 2020. That review looked at the approach taken to developing statistical models to award grades in the absence of exams that were cancelled because of the pandemic. Through this work, OSR established key factors that impacted on public confidence and identified these lessons to be useful for those developing models and algorithms in the future.

Applying our lessons learnt to QCovid®

We wanted to see if the lessons learnt from our review of the grading models in 2020 could be applied in a different context, to a different sort of algorithm, and test whether the framework stood up to scrutiny. We chose another model to carry out the testing, the QCovid® risk calculator, also developed in response to the pandemic.

In 2020, the Chief Medical Officer of England commissioned the development of a predictive risk model for COVID-19. A collaborative approach was taken, involving members from the Department of Health and Social Care (DHSC), NHS Digital, NHS England, the Office for National Statistics, Public Health England, University of Oxford plus researchers from other UK Universities, NERVTAG, Oxford University Innovations, and the Winton Centre for Risk and Evidence Communication. It was a UK wide approach agreed and including academics from Wales, Scotland and Northern Ireland.

The original QCovid® model that we reviewed calculates an individual’s combined risk of catching COVID-19 and dying from it, allowing for the inclusion of various risk factors. The QCovid® risk prediction model calculates both the absolute risk and the relative risk of catching and dying from COVID-19. The QCovid® model also calculates the risk of catching COVID-19 and being hospitalised but these results were not used in the Population Risk Assessment.

What is an absolute risk?: This is the risk to an individual based on what happened to other people with the same risk factors who caught COVID-19 and died as a result.

What is a relative risk?: This is the risk of COVID-19 to an individual compared to someone of the same age and sex but without the other risk factors.

The academic team, led by the University of Oxford, developed the model using the health records of over eight million people. It identified certain factors, such as age, sex, BMI, ethnicity and existing medical conditions, that affected the risk of being hospitalised or dying from COVID-19. The team then tested the model to check its performance using the anonymised patient data of over two million other people. Having identified these risk factors, NHS Digital applied the model to medical records of NHS patients in England and those identified as being at an increased risk of dying from COVID-19 were added to the Shielded Patient List (SPL).

This approach was ground-breaking as there was no precedent for applying a model to patient records to identify individuals at risk on such a scale. Before the development of QCovid®, the SPL had been based on a nationally defined set of clinical conditions and local clinician additions. People added to the SPL through application of the QCovid® model were prioritised for vaccination and sent a detailed letter by DHSC advising them that they may want to shield.

The QCovid® model was peer-reviewed and externally validated by trusted, statistical bodies such as the ONS and the results and the QCovid® model code were published.

What we found from reviewing QCovid®

In testing the lessons from the review of the grading models in 2020, we found that some lessons were not as relevant for QCovid®. For example, the lesson about the need for being clear and transparent on how individuals could appeal any decisions that the algorithm might have automatically made was less relevant in this review. This is because, although individuals were added to the SPL through the model, shielding was advisory only, and individuals (or GPs on their behalf) could remove themselves from the list. Finding lessons that were less relevant in a different context is to be expected as every algorithm or model will differ in its development, application, and outcomes.

As part of this review, we did identify one additional lesson. This concerned how often the underlying data should be refreshed to remain valid in the context of the algorithm’s use and appropriateness, especially if the algorithm is used at different points in time. This was not relevant for the review of grading models as they were only intended to be used once. However, in a different situation, such as the pandemic, where new information is being discovered all the time, this was an important lesson.

What do we plan next?

We found that the framework developed for the review of grading models proved to be a useful tool in helping to judge whether the QCovid® model was likely to command public confidence. It provided assurance about the use of the model and stood up well under scrutiny. Additionally, working on this review has helped us to understand more about QCovid® itself and the work behind it. QCovid® provides a great example that models and algorithms can command public confidence when the principles of Trustworthiness, Quality and Value (TQV) are considered and applied. In terms of how we will use these findings going forward, we have updated our algorithm review framework and this example will feed into the wider OSR work on Guidance for Models as it continues to be developed this year.

I really hope this work will be useful when we come across other algorithms that have been used to produce statistics and also that when we incorporate it into our Guidance for Models that others will benefit more directly too. So, this concludes my first blog in my Methods role at OSR, and in fact, my first blog ever!

Guest blog: Improving reporting and reducing misuse of ethnicity statistics

Richard Laux, Deputy Director, Data and Analysis, at the Equality Hub discusses his team’s work in improving reporting and reducing the misuse of ethnicity statistics in our latest guest blog, as part of the 30th anniversary of the United Nations’ Fundamental Principles of Official Statistics.

In my role as the Head of Analysis for the Cabinet Office’s Equality Hub I am in the privileged position of leading the team that analyses disparities in outcomes between different ethnic groups in the UK.

The reasons for disparities between ethnic groups are complex, and include factors such as history, relative levels of deprivation, the different age profile of some ethnic groups as well as many other factors. Despite the complexity of the issues, my team and I do all we can to prevent misuse of the data and help ensure that robust and clearly explained data are furthering the debate on race and ethnicity, which is an emotive topic for many people in this country.

My team’s responsibility for this is firmly rooted in the UN Principle 4 of preventing the misuse of statistics. We do this in a number of ways that align with this principle.

One way we do this is through bringing several analyses together to paint a broad-based picture of a topic of interest. For example, when supporting the Minister of State for Equalities on her reports on progress to address COVID-19 health inequalities we synthesised a large body of research describing the impact of the pandemic on ethnic minority groups. Much of this work involved my team reconciling and reporting on different sources and drawing robust conclusions from different analyses that didn’t always entirely agree.

A second way we try to prevent misuse of data is through the clear presentation of statistics, an example being Ethnicity facts and figures. This website was launched in October 2017 and since then it has been a vital resource to inform the debate about ethnicity in the UK. It gathers together government data about the different experiences of the UK’s ethnic groups and is built around well-established principles, standards and practices for working with data like the Code of Practice for Statistics.

We try to make the content on the website clear and meaningful for people who are not experts in statistics and data. It also contains detailed background information about how each item of data was collected and analysed to help those users with more interest or expertise in statistics draw appropriate conclusions.

The Commission on Race and Ethnic Disparities report recommended that RDU lead work to further improve both the understanding of ethnicity data and the responsible reporting of it (and thereby helping to prevent its misuse). As part of this work, we will consult on how to improve the Ethnicity facts and figures website, including whether we increase the amount of analysis on the site to help users better understand disparities between ethnic groups. Some of this might be in a similar vein to Office for National Statistics (ONS) work during the pandemic on ethnic contrasts in deaths involving the COVID-19. This modelling work showed that location, measures of disadvantage, occupation, living arrangements, pre-existing health conditions and vaccination status accounted for a large proportion of the excess rate of death involving COVID-19 in most ethnic minority groups.

Of course, there can be some difficulties with data that might lead to its misuse: datasets can vary greatly in size, consistency and quality. There are many different ways that ethnicity is classified in the datasets on Ethnicity facts and figures, and these classifications can differ widely depending on how and when the data was collected. For example, people might erroneously compare the outcomes for an ethnic group over time thinking it has remained the same whereas in fact it has changed; this might happen if someone is looking at data for the Chinese, Asian or Other groups over a long time period, as the Chinese group was combined into the ‘Other’ ethnic group in the 2001 version of the aggregated ethnic groups, but combined into the Asian group in the 2011 version of the aggregated ethnic groups in England and Wales.

We also try to minimise misuse and misinterpretation by promoting the use of established concepts and methods including information on the quality of ethnicity data. Our quality improvement plan and significant contribution to the ONS implementation plan in response to the Inclusive Data Taskforce set out our ambitions for improving the quality of ethnicity data across government. We will also be taking forward the Commission for Race and Ethnic Disparity’s recommendation that RDU should work with the ONS and the OSR to develop and publish a set of ethnicity data standards to improve the quality of reporting on ethnicity data. We will consult on these standards later this year.

Finally, we raise awareness and knowledge of ethnicity data issues through our ongoing series of published Methods and Quality Reports and blogs. For example, one of these reports described how the overall relative stop and search disparity between black people and white people in England and Wales can be misleading if geographical differences are not taken into account.

We have significant and ambitious programmes of analysis and data quality work outlined for the future. I would be grateful for any views on how we might further help our users in interpreting ethnicity data and preventing misuse.

Why migration statistics matter

Like many out there I wake up every morning hoping that a way, that protects life, has been found to bring peace in Ukraine. As it says in one of my young daughter’s books, “The world’s already far too full of cuts and burns and bumps”^[1].

Unfortunately, the conflict continues and people from Ukraine are fleeing. On 10 March 2022, the UN Refugee Agency (UNHCR) estimates that just over 2.3 million people have fled Ukraine since 24 February 2022. Across the UK there has been an outpouring of public sympathy for Ukrainian people forced to flee. The Government has introduced some new visa routes for Ukrainians; and debate continues among the public, in the media and in Parliament about whether the UK is doing enough to help.

As with any crisis, lots of decisions will need to be made. Decisions by individuals, by governments and by agencies and by organisations helping to support people flee Ukraine and build new lives. Data and statistics are a key part of this decision-making process. For example, to inform local and national emergency response planning, the Office for National Statistics (ONS) has published new data about the number of Ukrainian nationals by local authority and the Home Office has published the number of people applying for these new Ukrainian visa routes.

Here at the Office for Statistics Regulation (OSR) I lead on OSR’s work on migration. At the heart of my role is ensuring that data and statistics serve the public good. What does this mean in this context? It means ensuring that the best possible data are available to inform decision-making. And it also means ensuring data are publicly available to help the public understand the impact of decisions made, for example to evaluate the impact of new visa routes for Ukrainians and the impact this has on the make-up of society in the UK.

Earlier this month, we published the first in a series of reports looking at how the Office for National Statistics (ONS) is transforming the way it measures international migration. These statistics provide estimates of how many people are flowing into and out of the country from across the world and what the impact is on the number of migrants in the UK. The previous methods, based on the International Passenger Survey (IPS), had limitations so it’s great to see new robust methods being developed in a credible way and in discussion with expert users. Our report welcomes the ambitious and collaborative approach being taken by the ONS to transform the way it measures international migration and recommends some ways ONS can build on this good work. I would like to thank all those we have engaged with us as part of this work for their openness and time. I look forward to continuing this work to ensure that the transformed migration statistics are trustworthy, high quality and support society’s needs for information.

More widely we also engage with other government bodies responsible for the production and publication of statistics and data on migration. For example, we regularly engage with the Home Office, which is responsible for publishing a wide range of statistics about migrants. We have recently written to the Home Office about the publication of data on migrants arriving in Small Boats. In our letter we welcomed the Department’s plans to regularly publish additional data about this topic.

At the OSR we want our work to have an impact. That means ensuring that data and statistics are there to inform decision-making across society, the public, private and third sectors and to help hold organisations to account. This is at the heart of what I do as a statistical regulator at the OSR and at the core of our migration work. I just hope in a small way this can have a positive impact on what is happening out there in the world today.

If you would like to feed into any of our work on migration statistics please get in touch with Siobhan Tuohy-Smith.

[1] Donaldson J & Scheffler A, 2010, Zog, Published in the UK by Alison Green Books.