The Office for Statistics Regulation has today published a review of the approach to developing statistical models to award 2020 exam results. As the regulator of official statistics in the UK, it is our role to uphold public confidence in statistics.
Statistical models and algorithms used by government and other public bodies are an increasingly prevalent part of contemporary life. As technology and the availability of data increase, there are significant benefits from using these types of models in the public sector. We are concerned that public bodies are now less willing to use statistical models to support decisions in the future for fear of a public backlash.
To address this concern, the report identifies lessons from the approaches taken to awarding grades in 2020. These lessons apply to those that develop statistical models, policy makers who commission statistical models and the centre of government. Our aim with these lessons is to ensure that statistical models, whatever they are designed to calculate, can command public confidence in the future.
We argue that effective use of models is more than just a technical exercise in coding. We found that public confidence in statistical models is supported by considering three principles: be open and trustworthy, be rigorous and ensure quality throughout and meet the need and provide public value.
We also call on the centre of government, in collaboration with the administrations in Scotland, Wales and Northern Ireland, to improve the support, guidance and professional oversight for those working with statistical models and algorithms
In the exams case, we found that the public bodies responsible for designing the models all acted with integrity. We identified that there were inherent challenges in the task which made it difficult to deliver exam grades in a way that commanded public confidence. We conclude that many of the decisions made supported public confidence, while in some areas different choices could have been made. In particular, we consider that they could have done more to convey publicly the limitations to the use of models to award grades.
Ed Humpherson, Director General for Regulation, said:
“Public debate about algorithms veers between blind optimism in their potential and complete rejection. Both extremes have been very evident, at different times, in the discussion of the use of models to award exam grades. Our report offers a more balanced perspective: algorithms and broader model-driven decisions can operate in the public interest. But only when there is a rounded approach that goes beyond the technical coding – and considers the public impact of the models and is open about their limitations”.
Notes to Editors:
- The Office for Statistics Regulation (OSR) is the regulatory arm of the UK Statistics Authority. We provide independent regulation of all official statistics produced in the UK assessed against the Code of Practice for Statistics.
The approaches to awarding grades were regulated by four bodies:
- In England, Office of Qualifications and Examinations Regulation (Ofqual)
- In Scotland, Scottish Qualifications Authority (SQA)
- In Wales, Qualifications Wales
- In Northern Ireland, Council for the Curriculum, Examinations & Assessment (CCEA).
For more information, please contact Suzanne Halls on 07411 212300, or email suzanne.halls@statistics.gov.uk