UK Statistical System UK Statistics Authority Code of Practice

Six steps to make your model a role model

My favourite number is 6. It was the number Roberto Carlos, my footballing idol, wore when he played for Brazil. As a kid I remember trying to recreate his infamous banana free kick versus France (1997) over, and over again. Practice makes perfect they say. So here, in honour of Senhor Carlos, I am going to provide 6 pieces of advice from our latest publication Guidance for Models so you can all practice turning your models, into a role models. If you just want to know why we made this guidance, skip to the end of this blog. 

Every year we throw away 2.5 billion takeaway cups in the UK

Don’t create your model using the same criteria as a takeaway disposable cup: use once and throw away. A model, or components of it, should be reusable. Reusable not just by you, but by others too. It should also be robust (see testing) and adaptable (see modular). Do not make a disposable cup just for coffee, if someone else is also just for tea, when you can work together to make a sturdy cup for both tea and coffee.

Weed your borders

Like beautiful garden borders, models need maintenance. Borders may need mulching in the winter, pruning in the spring and lots of watering in the summer. Likewise, model dependencies may need updating, bugs resolving, and code changes made as a result of changes to data schemas. Teams need to plan their time accordingly to make sure the model still works and remains fit for purpose, especially as others may rely on your model’s outputs for their own model input.

Don’t be like a tragic episode of Grand Designs

We’ve all seen a Grand Designs episode when the keen, wannabe architect wants to design, build and project manage on their own dream home. It often ends in disaster: over budget, late, relationship breakups, and a lot of stress. Likewise, you need the correct people involved at the correct time to make your model a success. Model design plans should be scrutinised and checked. Experts should be consulted early on, instead of after things go wrong. Lastly, like all house builds, the model should be verified and signed off by someone you trust to make sure it is safe and secure. 

It took Jean-François Champollion years to decipher the Rosetta Stone

Model documentation should be accessible and understandable to a wide range of audiences. You should not just publish detailed, technical documentation as not everyone will be able to understand the purpose or scope of your model. This may lead to your model being misused. You should explain your model as best as you can. If the nature of the model means it is hard to explain, you should describe how users can interpret your model and its outputs at the very least. Your should open source your code if possible, and provide installation guides, code comments and examples as well. 

Be as analytical in choices as you are on Netflix

Aim to be as detailed in your model decisions as you are when choosing your next TV show to binge watch. Like choosing between a horror and a rom-com, you must understand what kind of model you need based on your scope and aims. You should seek advice and guidance from experts. Like the impossibilities of trying to stay up to date with the latest shows using a 90’s copy of the Radio Times, your decisions should be based on relevant and up to date information. Lastly, don’t overcommit and carry on if things don’t seem to be going well. Use regular checkpoints to reassess against your original needs. No one wants to force themselves through another experience like season eight of Game of Thrones. 

Be as ethically minded as you are when switching your energy supplier 

You read an article last week about the environmental benefits of veganism and want to give it a go? Great. You switched your electricity supplier to a 100% renewable electricity supplier? Way to go. You stopped going to that pub that treated its workers poorly? Power to the people! Now also understand that data, design choices and model selection all can have ethical implications. Power can be given, or taken, from certain groups based on the models we create and use. Ethics should not just be a tick box exercise; it should be the cornerstone of your model design and development.

Sure, nice analogies, but why did you actually create this guidance?

Last year the global pandemic thrust us into the limelight following a series of high profile uses of statistics for decision making. One of the biggest pieces of work we did last year was our review of the approach to developing statistical models to award 2020 exam results. “Algorithms” were blamed, with one headline stating “Dreams ruined by an algorithm” (BBC NI website). As such, we have been concerned about the threat of undermining public confidence in statistical models more broadly. 

That exam review work took us into new frontiers by commenting on the use of statistical models that influence public life, not just those that produce official statistics. But statistical models are just one range of tools used by government analysts, data scientists and statisticians. Increasingly, newer techniques such as machine learning (ML) are being tested and deployed in the production of statistics and used to inform decisions. Furthermore, with the creation of the Office for Artificial Intelligence and the Government’s National AI Strategy, we are likely to see an increased use of more advanced Artificial Intelligence (AI) techniques going forward.  

As a result, we identified this as a crucial time to provide guidance for the use of models, regardless of whether they are statistical models, machine learning models or AI models. There have been a number of publications for ethical guidance for models (Ethics, Transparency and Accountability Framework for Automated Decision-Making, Data Ethics framework) as well as the creation of the UK Statistics Authority’s Centre for Applied Data Ethics. There are also a number of technical guides on how to develop models (Aqua Book). However, we saw that there was no current guidance that suitably brought together social, ethical and technical aspects for all elements of model creation: data, design, development, delivery and deployment.  

We believe our role as a regulator, and our experience of the exam review from last year, puts us in a prime position to provide this socio-technical guidance for models. As a result, we have published our alpha release version of our model guidance with the aim to obtain feedback and comments from a wide range of users. 

If you have any feedback, please get in touch! We aim to release an updated version of the guidance in early 2022. 

Transparency: How open communication helps statistics serve the public good

Over the past 18 months we’ve talked a lot about transparency. We’ve made public interventions such as our call for UK governments to provide more transparency around COVID data, and it’s been prominent in our vision for the future of analysis in government, including in our Statistical Leadership and State of Statistical System reports.

But what do we mean when we talk about transparency? Why do we care? And what can be done to support it?

What do we mean by transparency?

Transparency is about working in an open way. For us, transparency means being open about the data being used. Explaining what judgements have been made about data and methods, and why. Being clear about the strengths and limitations of data – including what they can tell us about the world, and what they can’t. It also means making sure data and associated explanations are easy to find and clearly presented. It is at the core of many of the practices outlined in the Code of Practice for Statistics.

Why does it matter?

The pandemic has increased the public appetite for data and drawn attention to the significance of data in decision making. Many of us will have become familiar with the phrase “data, not dates” – a phrase which UK government used as it set out its road map for easing coronavirus restrictions. In a context when so many have been asked to give up so much on the basis of data it is especially important that the data are understood and trusted. Transparency is essential to this.

Transparency supports informed decisions. Appropriate use of data is only possible when data and associated limitations are understood. We all make daily decisions based on our understanding of the world around us. Many of these are informed by data from governments, perhaps trying to understand the risk of visiting a relative or judging when to get fuel.

We also need this understanding to hold government to account. Clearly presented data on key issues can help experts and the public understand government actions. For example, whether the UK is taking appropriate action to tackle climate change? Or how effectively governments are managing supply chains?

Transparency gives us a shared understanding of evidence which supports decisions. It allows us to focus on addressing challenges and improving society, rather than argue about the provenance of data and what it means. It supports trust in governments and the decisions they make. It allows us to make better individual and collective decisions. Ultimately, it ensures that statistics can serve the public good.

What is government doing?

We have seen many impressive examples of governments across the UK publishing increasingly large volumes of near real time data in accessible ways. One of the most prominent being the coronavirus dashboard and equivalents in other parts of the UK, such as the Northern Ireland COVID-19 Dashboard.

It has become routine for data to be published alongside daily Downing Street briefings, and through its additional data and information workbook Scottish Government has put in place an approach which enables it to release data quickly when necessary. We have also seen examples of clear explanations of data and the implications of different choices, such as the Chief Statistician’s update on the share of people vaccinated in Wales.

However, this good practice is not universal. Transparency regularly features in our casework. We have written public letters on a range of topics including Levelling Up, fuel stocks, hospital admissions and travel lists. We want to see a universal commitment to transparency from all governments in the UK. This should apply to data quoted publicly or used to justify important government decisions. Where data are not already published, mechanisms need to be in place to make sure data can be published quickly.

The Ministerial Code supports this ambition by requiring UK Government ministers to be mindful of the Code of Practice for Statistics – a requirement that is also reflected in the Scottish and Welsh Ministerial Codes and the Northern Ireland Guidance for Ministers. In response to a recent Public Administration and Constitutional Affairs Committee report the UK Government itself said:

“The Government is committed to transparency and will endeavour to publish all statistics and underlying data when referenced publicly, in line with the Code of Practice for Official Statistics.”

What is OSR doing?

We want to see statistics serve the public good, with transparency supporting informed decisions and enabling people to hold government to account. Over coming months, we will:

  • build our evidence base, highlighting good examples and understanding more about barriers to transparency.
  • continue to intervene on specific cases where we deem it necessary, guided by the UK Statistics Authority’s interventions policy.
  • work with external organisations and officials in governments to support solutions and make the case for transparency.

What can you do?

We’re under no illusion: OSR can’t resolve this on our own. Whether an organisation or individual we need your help.

You can question the data you see. Does it make sense? Do you know where it comes from? Is it being used appropriately?

You can raise concerns with us via regulation@statistics.gov.uk – our FAQs set out what to expect if you raise a concern with us. We’d also love to hear from other organisations with an interest in transparency.

And you can keep up to date with our work via our newsletter.

 

 

Reflections on lessons learned from COVID for health and social care data

You may have noticed that the last 18 months or so have been rather unusual. In fact it’s getting difficult to think about what things were like before masks, distancing and the universal smell of alcohol gel.  And there’s another change to which we have become accustomed – the daily parade of statistics, the use of graphs on the news, and the huge presence of scientific and statistical discussion, both in the media and among ordinary people who are not even statisticians!

The scale and ambition of the health data being made available would have been unthinkable just two years ago, as would be the complexity and sophistication of the analyses being conducted. But the Office for Statistics Regulation’s ‘Lessons Learned’ report argues that we should not be complacent: we need to press harder for more trustworthy, better quality, and higher value statistics.

There are a few recommendations that stand out for me. First, Lesson 9 focusses on improved communication. Back in May 2020 I stuck my neck out on the Andrew Marr show and criticised the press briefings as being a form of ‘number theatre’, with lots of big and apparently impressive numbers being thrown around without regard for either accuracy or context. This attracted attention (and 1.7m views on Twitter). But although some dodgy graphs continued to appear, the presentation of statistics improved.  Crucial to communication, however, is Lesson 1 on transparency – it is essential that the statistics underlying policy decisions, which affect us all, are available for scrutiny and are not cherrypicked to avoid those that might rock some political boat. This requires both constant vigilance, and appropriate clout for professional analysts.

Lesson 7 deals with collaboration, reflecting the extraordinary progress that has been made both in collaboration across governments and with academic partners, all of whom have shown themselves (against archetype) to be capable of agile and bold innovations. The Covid Infection Survey, in particular, has demonstrated both the need and the power of sophisticated statistical modelling applied to survey data. Although of course I would say that, wouldn’t I, as I happen to be chair of their advisory board, which has enabled me to see first-hand what a proper engagement between the ONS and universities can achieve.

Finally, Lesson 3 addresses the idea that data about policy interventions should not just enable us to know what is happening – essentially ‘process’ measures of activity – but help us to evaluate the impact of that policy. This is challenging; Test and Trace has come in for particular criticism in this regard. For statisticians, it is natural to think that data can help us assess the effect of actions, with the randomised clinical trial as a ‘gold-standard’, but with an increasing range of other techniques available for non-experimental data. Again there is a need to get this up the agenda by empowering professionals.

An over-arching theme is the need for the whole statistical system to be truly independent of political influence from any direction. While this is enshrined in legislation, a continued effort will need to be made to make sure that work with data lives up to the standards expressed in the Code of Practice for Statistics, in terms of trustworthiness, quality and value. The pandemic has shown how much can be achieved with the right will and appropriate resources, and OSR’s ‘Lessons Learned’ point the way forward.

 

David Spiegelhalter is a Non-Executive Director of the UK Statistics Authority, which oversees the work of the Office for Statistical Regulation.

Launching our consultation: We want your views

As a regulator, we want to be as helpful as we can to producers of statistics to enable the release of valuable information, while also setting clear standards of practice. In the pandemic we supported producers by granting exemptions to the Code of Practice for Statistics to enable some statistics to be released at times other than the standard time of 9.30am.

Market sensitive statistics could no longer be released after the usual lock-in briefings, so we agreed for them to be released at 7am. This has meant that the lead statisticians have been able to speak in the media and explain the latest results.

We also enabled some statistics related to COVID-19 to be released later in the day as soon as they could be shared publicly with sufficient quality assurance. It has meant for example that both the Coronavirus infection survey bulletins and the Public Health Scotland COVID-19 weekly report for Scotland are released at noon.

Having a specific time of release has helped ensure consistency in release and grow confidence that official statistics are truly independent of political preferences and interference. The pandemic has brought to light how important timely statistics are, and the huge demand has meant that release timings have had to change so that the statistics remain relevant and useful to the public.

As we look beyond the pandemic, we have been considering whether we should amend the Code of Practice to enable more flexibility for producers but at the same time keep the benefits of consistency and protection against interference. We are grateful to everyone who has shared their views with us in our discovery phase. It has helped us consider a range of issues.

Consultation

We are pleased to announce that a 12-week consultation will begin on 28 September 2021, ending on 21 December 2021. Our consultation paper will set out some proposals on release approaches that look to maintain the benefits of standard release times but also support some greater flexibility. The Authority will carefully consider the responses before deciding on its preferred release practice.

We encourage you to consider the suggestions and to share your views with us.

A closer look at loneliness statistics

At OSR, we have always been aware of the importance of loneliness statistics on a national and local scale. In 2019, we started a systemic review of loneliness statistics to investigate the state of official statistics on loneliness in the UK. 

Initially, we found there were some significant gaps in loneliness data that were not being filled by official or national statistics. Statistics users we spoke to, such as charities focused on loneliness, told us this made it more difficult for them to carry out their core functions of preventing and tackling loneliness among the UK population.  

We heard that good quality statistics that covered local and regional geographies were needed in order for them to deliver their services, allocate funding, and in some cases, present evidence to their regional parliaments. Where official statistics were not meeting these needs, expert users were often stepping in and producing their own statistics to fill data gaps. Given this, we identified a range of specific recommendations to help improve official statistics on loneliness. 

Like many pieces of work during this period however, the pandemic made us re-think our approach. The pandemic has changed how we all think and act, including how we think about loneliness. Understanding and addressing loneliness among the population has become a focus for governments and policy makers. In response, statistics producers have had to develop their loneliness statistics to meet society’s need for information. As a result, many good developments have happened in this area and we’ve found that statistics producers have been filling in some of the key gaps we identified when we first started looking at these statistics. Our new report published today commends the efforts by statistics producers in creating statistics that better serve the public good in answer to these societal changes. 

This isn’t to say that improvements can’t still be made though. Users we spoke during the pandemic still identified some key gaps in the official statistics landscape on loneliness. We would encourage statistics producers to build on the work they had achieved in the last 18 months and to continue to take forward producing statistics that meet user needs and offer value for charities and academics in preventing and researching loneliness.  

Continuing the loneliness review was one of the first pieces of work I got when I started my placement year at the OSR last September. I’ve really enjoyed working on the report and having the opportunity to lead a review and conversations with producers. Seeing the report published on my last day at the OSR brings a wonderful and rather cyclical end to my year! The work isn’t ending with me though. As an organisation, we are looking forward to continuing working in this area and assisting producers to develop their statistics to better meet user needs. If you would like to contact us about this, please email my colleague, Emma Harrison. 

“Welp. We screwed up”: A lesson in admitting your data mistakes

A couple of months ago this tweet from the Governor of Utah caught my eye:  

The background was that the Utah Department of Health had identified an error in its published vaccinations figures – fewer people had been vaccinated against coronavirus than had been reported. In a public letter to his fellow Utahns, Governor Cox admitted the mistake.  

Here at the Office for Statistics Regulation we love this kind of openness and action. Five things stood out to us from Governor Cox’s letter, which we think all statistics producers in the UK should be thinking about when it comes to data errors. 

  1. Identify your mistake

You can’t fix what you don’t know is broken. This is why rigorous quality assurance is so important. Statisticians need to regularly look under the hood of their analysis to assure themselves and others that it is correct. In Utah, a healthy dose of scepticism about an unexpectedly high vaccination rate made the data team double-, triple- and quadruple-check their figures, until they found the mistake. So, as a statistics producer ask yourself: how do I assure myself that my analysis is correct? Do the numbers look as I expect, and why or why not? What are the risk points for errors? If the root cause of an error isn’t obvious then it can help to ask the five whys until you reach it. 

  1. Be open

One of the things that impressed us most about this example was how direct and open the communication of the error was. There was clear ownership of the mistake and a willingness to correct the figures quickly, publicly and with humility. In the UK, our Code of Practice for Statistics requires government statistics producers to handle corrections transparently. It’s also important that government officials and minsters who use statistics are open about mistakes. 

  1. Fix it

Of course, once you have identified the mistake, it needs to be fixed. As well as being transparent about corrections, the Code of Practice asks that they are made as quickly as is practical. 

  1.  Explain

In Utah, Governor Cox explained that while they had reported that 70% of adults had received at least one dose of a coronavirus vaccine, the actual figure was 67%. In a separate statement, the Utah Department of Health went into more detail about the numbers and the mistake. Statistics and data should help people understand the world around them. So, when admitting a data error, it’s important to clearly explain the impact of it – what has changed and what does this mean? 

  1. Improve

The last, but perhaps the most important, step is to learn from the mistake – so that you can avoid it, or something similar, happening again. In Utah, the data team re-examined their processes to prevent this general type of error from being repeated. Statistics producers should reflect on why a mistake was made and what can be done to avoid it in future – and then share what they have learned, and what action they are taking, with their users. 

Statistics and data produced by governments serve a vital purpose. They keep us informed, help us make decisions and allow us to hold our governments to account – so we must have confidence in them and the people who produce them. As Governor Cox said, “trust consists of two things: competence and ethical behaviours”. We agree. The Code of Practice places a strong emphasis on trustworthiness. We see that trustworthiness is demonstrated by organisations which are open, and where people who work on statistics are truthful, impartial, and appropriately skilled. We are all human, we mess up and we make mistakes – but we can retain trust by actively looking for our mistakes, being open when we find them and by learning from them.  

Data makes the difference

This is a guest blog from Jonathan Smyth, Head of Communications and Fundraising at Action Mental Health.

As an organisation, Action Mental Health has long campaigned for better mental health services in Northern Ireland. Alongside partners in the sector, a key part of our campaigning included calls to produce a fully costed and properly resourced mental health strategy that would deliver real change for people in Northern Ireland. We were the only region of the UK without such a strategy despite being the region with the most need, something borne out by the fact that we have the highest prevalence of mental health problems in the UK.

In June 2021 then, we very much welcomed the announcement by Northern Ireland’s Health Minister – Robin Swann, MLA, of Northern Ireland’s first ever Mental Health Strategy, a ten-year vision that outlines three key themes encompassing 35 actions, as well as recognising the need to invest £1.2bn over the life time of the strategy to deliver its recommendations.

In addition to the new strategy, we very much welcome OSR’s in-depth review of mental health statistics in Northern Ireland, which has confirmed that existing statistics do not meet current user needs and sets out expectations in this area to make real change.

Across the many discussions and interactions, we have had, and continue to have with other mental health campaigners and professionals, one of the key things we hear is frustration at the lack of robust data and statistics around mental health and mental health service delivery in Northern Ireland. Given the obvious pressures on the health budget due to Covid it is vital that precious investment is not wasted and unfocused due to incomplete or false data.

We hear regularly from professionals about the challenges they face in navigating Northern Ireland’s fragmented services, which are often entirely different from area to area, or maybe they are simply described differently depending on postcode.

We’re also aware of the impact this has on our clients and the confusion and frustration it causes as they have to re-tell their story to many different healthcare professionals.

With this differentiation in service delivery comes issues with data collection – there is very little standardisation of data, across what is such a relatively small area, both in terms of geography and population. How then do we plan for better services and better outcomes if we don’t know what we are comparing from area to area? As an organisation trying to develop innovative new projects it is frustrating that there is no easily accessible source of data to ensure our valuable resources are properly focused on client need.

The lack of robust statistics in such a complex area can also present challenges in the digital age when misinformation can be spread so easily. Being able to vigorously challenge potentially damaging or worrying claims with evidence based, factual information is vital to protect public confidence and support public health messaging.

Our anecdotal evidence is supported by the findings of the newly published OSR (Office for Statistics Regulation) review of Northern Ireland’s mental health statistics which found:

  • The scarcity of robust mental health data in Northern Ireland has hindered the development of statistics and led to significant data gaps.
  • The lack of regional standardisation and a fragmented technology infrastructure has led to poor data quality, with limited consistency or comparability across the region.
  • Users find it difficult to locate official statistics across a dispersed landscape. Data accessibility could be improved.

In many ways these issues will be a fundamental challenge to the successful delivery of the new Mental Health Strategy. We need timely and robust data to underpin everything we do.

As that famous old business consultancy cliché goes:

“What gets measured gets done”

We have a unique opportunity with the new strategy in Northern Ireland to change how we support those with mental health issues, and robust and reliable data that targets investment and ensures better outcomes must be our goal.

You can find out more about Action Mental Health’s work by visiting our website or follow us on Twitter.

The Code pillars: Value – bringing something to the party

Value for me is about why it all matters. Value means that statistics and data are useful, easy to access, remain relevant, and support understanding of important issues. These things mean that the statistics will be used. Without statistics being of value, they risk becoming irrelevant.

Statistics should bring something to the party.

But why should we be invited to the party in the first place?

The pandemic has demonstrated the crucial value of statistics and of statisticians being involved in decision making and debate.

The statistical community really stepped up in providing new, innovative and highly relevant analyses. We sought to provide answers to the questions that decision makers and society needed answered. We saw the power of statistics to inform, to paint a picture. That is what value is all about. We were at the heart of the party! Without us being there, statistics cannot serve the public good. We need to value ourselves, and our statistics to demonstrate that value.

The value of valuable statistics

Early in my career, my colleague and I organised a session about government statistics for a school ‘careers in maths’ day. We were going to spend all day talking to 14-year olds about maths – probably not the most exciting prospect to them.

To try to engage them we developed a session based around why (government) statistics are important. The session, called ‘King for a Day’, involved crowning a child king (or queen) and getting the children to develop the list of statistics that they would need to run the kingdom.

Nothing was out of bounds. If the children decided that knowing the number of goals scored by the top football teams was a priority for their king, then it went on the list. I didn’t realise it at the time, but we had decided to talk about the value of statistics rather than simply how to produce good quality ones.

Without valuable statistics, the children realised their kingdoms couldn’t run properly, and their ‘citizens’ couldn’t hold them to account. They learned the value of valuable statistics, and why they are essential for the public good.

So how do we ensure that we are invited to the party?

We keep getting invited by….

  1. Being relevant – engaging in conversation with others at the party, listening, understanding what they need and responding accordingly.
  2. Being accessible – recognising that different party goers need different things to get the most from the party.
  3. Being clear and insightful – clearly explaining to the others what we are bringing to the party and how it can be useful. Ensuring that what we bring compliments what others are bringing.
  4. Being innovative –keeping listening and improving what we bring.
  5. Being efficient – recognising that we can share resources. Providing a clear rationale for why we are asking for certain things to be supplied to the party, and not overburdening others by asking them to contribute too much.

In short, following the Code of Practice for Statistics, and adhering to its three pillars; Trustworthiness, Quality and Value, ensure that statistics serve the public good.

For more information on the Code and the three pillars, you can visit the Code website. There are also case studies that demonstrate how statistics producers have implemented different practices in the Code.

 

 

 

 

 

 

Empowering statistical leaders

As part of our work on statistical leadership, we are hosting a series of guest blogs. This blog is from Stian Westlake, Chief Executive at the Royal Statistical Society.

From remote working to online shopping, the pandemic has been a great accelerator of long-term shifts. It has done much the same thing to the role of data and statistics within public life.

This was really brought home to me in late 2020 when the RSS’s panel of distinguished experts sat down to decide our Statistic of the Year. In past years, our choice had usually generated a quirky news piece, intended to highlight how statistics could make sense of the big stories of the year. But when we looked through the nominations for 2020, we realised things were different: the statistics before us were the big stories of the year. The news of the pandemic, its spread, and its impact on lives and on society were being understood through the medium of statistics.

In much the same way, statistics became a central tool of our collective efforts to understand and to tackle Covid. Crucial projects like the Coronavirus infection survey, the UK Coronavirus dashboard, and the RECOVERY trial were as central to the UK’s Covid response as Nightingale hospitals or the vaccine procurement programme, and each was, in its different way, an exemplary statistical undertaking. Statisticians were in demand across government, and proved their value time and again.

So it is extremely timely that earlier this year the Office for Statistics Regulation published its major review, Statistical leadership: making analytic leadership count. Others have written eloquently about several of the main themes of the report, such as the importance of statistical skills, and of transparency and trust. While these are dear to the RSS’s heart, there is another theme in the report that I think is especially important: empowering statisticians to provide leadership, and ensuring they have strong career prospects.

One way of thinking about the crucial importance of empowered statisticians is to consider the counterfactual. What happens when if the other conditions for statistical leadership – such as technical skills and transparency – are met, but if organisations fail to give statisticians the right organisational roles, access and opportunities?

When this happens, we see a very particular failure made. Statisticians are left out of the loop of strategic planning. Data is seen as a specialist function to be commissioned as an afterthought, often to justify rather than inform a decision. And the commissioning process breaks down: statistical projects are assigned by leaders with limited statistical background, sometimes with unrealistic objectives and little chance to iterate during the project. The near-term results are projects that are frustrating to work on and disappointing for users. The longer-term results is that skilled statisticians are demoralised and drift away. We’ve all seen organisations like this; we may even have worked at some. Sometimes statistics or statisticians get the blame, and we hear talk of mutant algorithms or statistical errors. But the root cause isn’t in the data or the methodology: it is a problem of organisation.

But the good practice of the past eighteen months have shown to the world at large that there is a better way. In our experience, this relies on a few elements.

First of all, putting statistics, data and evidence at the heart of the organisation’s strategy. This means those senior leaders who aren’t statisticians gaining the skills to be users of statistics and to work well with statisticians, and statisticians being supported and trained to take senior leadership roles, rather than existing as a permanent auxiliary function. This helps make statistics and data intrinsic to the organisation’s workings.

Secondly, it requires strong career development opportunities for statisticians. Technical skills are important, but for true statistical leadership these need to be complemented with opportunities to learn general management and other operational skills. Crucial to this is mentorship. (This is why the RSS runs a mentorship scheme for candidates for our Chartered Statistician designation.) One of the silver linings for some statisticians of the immense workload imposed by the pandemic has been the exceptional opportunities to try out new roles in other organisations, as statistical and data skills have been at such a premium. Wouldn’t it be good if the volume and quality of these opportunities continued once the burden of Covid-related work has subsided?

Thirdly, it requires managers and heads of profession to be mindful about the make-up of the profession and to ensure it is open, diverse and growing. Research has shown that lack of opportunity and diversity is a big barrier to society’s scientific potential; it is likely that the same holds true for our discipline. This means redoubling our efforts to increase the diversity of the statistical workforce when it comes to protected characteristics. It also means promoting non-traditional routes into the profession, building on the GSS’s apprenticeship and degree apprenticeship scheme, and making the most of in-work skills schemes like the RSS’s Data Analyst, Graduate Statistician and Chartered Statistician designations, and the competency framework we have designed for them.

Getting these vital human-level, organisational questions right is essential for a thriving statistical profession. And that in turn is indispensable for anyone who cares about rigorous, useful, trustworthy statistics.

The Code Pillars: Quality

When I joined OSR as a placement student last September, the Code of Practice for Statistics was unknown territory. It certainly sounded official and important. Was it password protected? Would I need to decipher something or solve a puzzle to get in?

It soon became clear to me that this elusive ‘Code’ was at the heart of everything I would be doing at OSR. Not wanting to remain in the dark any longer, I dutifully dragged it to the front of my bookmarks bar and began to familiarise myself with its contents. (Thankfully no complicated code-cracking required).

The Trustworthiness and Value pillars appeared to be pretty straightforward. Yet, something about the Quality pillar didn’t seem quite so inviting. It sounded like the technical, ‘stats-y stuff’ pillar, that my degree background in economics and politics would surely leave me ill-equipped to understand.

*Spoiler alert* I was wrong.

It turns out that ensuring statistics are the highest quality they can be, isn’t as complicated and technical as I once feared. Quality simply means that statistics do what they set out to do and, crucially, that the best possible methods and sources are used to achieve that.

There are lots of ways that statistics producers can meet these aims. For example, quality can be achieved through collaboration. This can be with statistical experts and other producers, to arrive at the best methods for producing data. It can also be with the individuals and organisations involved in the various different stages of the production process – from collecting, to recording, supplying, linking, and publishing. Collaborating in these ways not only helps to ensure that statistics are accurate and reliable, but also that they are consistent over time and comparable across countries too.

There are lots of other important-sounding documents like our Code of Practice that set out national or international best practise and recognised standards and definitions for producing statistics and data such as the GSS harmonisation standards and the Quality Assurance Framework for the European Statistics System. These also help producers ensure that their statistics and data meet the highest possible standards of quality.

Quality is not only important at the producer-end of the equation, but at the user-end too. It is vital that producers are transparent with their users about how they are ensuring the quality of their statistics. This means telling users about the steps they take to achieve this, and being clear with them about the strengths and limitations of the statistics with respect to the various different ways in which they could be used.

For an indication of just how important quality is, the Quality Review of HMRC Statistics we conducted last year is a prime example. After identifying an error in its published Corporation Tax receipt statistics, HMRC asked us to assess its approach to managing quality and risk in the production of its official statistics. With the Code as our guide, we were able to review HMRC’s existing processes and identify potential improvements that could be made to reduce the risk of statistical errors in the future.

This is just one example of how high-quality data fulfils our vision of statistics that serve the public good. We have found many others across our work and we continue to support producers to consider quality when producing statistics. Last year, we published new guidance for producers on thinking about quality, which was inspired by the HMRC review and the questions we asked.

If you’re interested in finding out more about Quality and the other pillars of our Code, check out the Code of Practice website. I promise it’s not as scary or elusive as it sounds…

 

Did you know we have case studies on our Code website too? Here are some of our examples that highlight examples of good practice in applying the quality pillar of the Code.

  • Q1 – Ensuring source data is appropriate for intended uses
  • Q2 – Developing harmonised national indicators of loneliness
  • Q3 – Improving quality assurance and its communication to aid user interpretation