OSR’s Head of Data and Methods, Emily Barrington explores the work taken to deploy artificial intelligence and statistical modelling within government while adhering to a high regulatory standard
Thinking back to when I joined the Office for Statistics Regulation (OSR) in late 2019 (just before the pandemic hit), Artificial Intelligence (AI) and other complex statistical modelling was still in its infancy within government. There were pockets of work being done here and there and guidance was being produced but there was nothing public facing, and nothing to help analysts understand how to organise model development that could help instil trust from the public’s perspective.
At this point you may be thinking, but aren’t you the regulator of statistics? Why are you thinking about AI models? Well, two reasons. Firstly, it comes down to definition. AI is the new buzzword but when you strip it back to its core components it’s really just complex statistical modelling (albeit on a larger scale and with bigger computers!) so any guidance that would apply to statistical modelling will also apply to AI and vice versa. Secondly, helping build public trust is in our ethos and, when it comes to AI use within government, the outputs of such models often have a public impact – be it directly or indirectly through policy change.
Not long after I joined I started looking at how we, at OSR, could have a voice in this area to champion best practice through our pillars of Trustworthiness, Quality and Value (TQV).
The pandemic effect
If anything, the need for data and insight throughout the pandemic helped break some of the barriers that had been stopping AI/complex modelling taking off within government. Things like data sharing and public acceptance of use has generally been greater during the pandemic which may have been driven by the need to help save lives. This drive, however, sometimes led to misjudgement and this is what happened when awarding exam grades in 2020 and led to our review on ‘Securing public confidence in algorithms’. This was the first time OSR had worked on anything related to algorithms so specifically and the lessons that were drawn from the work resonated well, people thought we had something to give – and we agree with them!
This work also made us think outside the box when it came to the Code of Practice for Statistics (The Code). After all, the model used to award exam results was not official statistics, neither was it AI for that matter, but the Code still helped us when making our judgements.
Back to championing best practice
By the time the review on awarding exam results was published, we had already started putting down some thoughts on how the code could be applied when using models and later that year our alpha version of ‘Guidance for Models: Trustworthiness, Quality and Value’ was published. It was published as alpha because we wanted to get as much feedback as possible before promoting more widely – this was our first time in this space after all. We also felt there might be a better way to present the messages but needed some further thought and input from the wider analysis and data science communities.
The pillars of Trustworthiness, Quality and Value (TQV)
Since the publication of the alpha guidance, we have come a long way in thinking about what the Code and its pillars really embody when broken down and have matured our thinking on statistical modelling. Today we published our finalised version of ‘Guidance for models: Trustworthiness, Quality and Value’ which takes the TQV messages and brings them to life for model planning and development. We have softened our focus on the Code principles since the alpha version and taken a step back to concentrate on the most important Code considerations for public good of models. This came from feedback from analytical and data science communities that the messages are stronger when not linked to the Code specifically. We have also incorporated all the lessons from our review on Securing public confidence in algorithms’ and our follow-up case study on QCOVID.
We now have a guidance which we believe embodies what is needed to help build public confidence and trust when deploying statistical models. But I guess the proof is in the pudding…
Thoughts?
If you have any feedback, thoughts or use cases where you found our guidance helpful please do not hesitate to contact OSR Data and Methods – we’d love to hear from you!