OSR review of approach to developing statistical models designed for awarding 2020 exam results

The Office for Statistics Regulation (OSR) is undertaking a review focused on the process of developing the statistical models designed for awarding 2020 exam results.

The review aims to identify lessons for public bodies considering the use of statistical models to support decisions. The lessons will be based on our consideration of the extent to which the qualifications’ regulators across the UK developed their models in line with the principles set out in the Code of Practice for Statistics.

Guided by the pillars of the Code - Trustworthiness, Quality and Value, we will initially be exploring the following two questions.

To what extent did the approach to developing and communicating the models support public confidence in the outputs?

This will consider clarity of purpose, governance, transparency, communication, ethics and public acceptability of the algorithms as well as the role of the appeals process.

To what extent were the algorithms developed appropriate for their intended purpose?

This will include the model selection and limitations​, quality of the data sources, quality assurance of outputs​, innovation and collaboration during the model development and validity of claims around the algorithm.

It is not within OSR’s role to regulate operational decisions made by government bodies about outcomes for individuals, nor to form a judgement on decisions about government policy. As such we will not review the implications of the models for the results of individual students or evaluate policy decisions on the most appropriate way to award exam grades in the absence of exams.

To support us with the review we have set up an Expert Oversight Group which is meeting on a regular basis. The purpose of the group is to provide technical oversight to the OSR project team. Members of this group are National Statistician Professor Sir Ian Diamond, Professor at Imperial College London, David Hand, Chair of the UK Statistics Authority, Sir David Norgrove and leading statistician and Chair of the Winton Centre at Cambridge University, Professor Sir David Spiegelhalter.

We aim to publish findings at the end of October.

These webpages will be updated as the review progresses.

If you wish to contact us regarding this review please email regulation@statistics.gov.uk 

Correspondence:

Ed Humpherson to Philip Blaker: OSR review of approach to developing statistical models designed for awarding 2020 exam results

Ed Humpherson to Justin Edwards: OSR review of approach to developing statistical models designed for awarding 2020 exam results

Ed Humpherson to Roger Taylor: OSR review of approach to developing statistical models designed for awarding 2020 exam results

Ed Humpherson to Fiona Robertson: OSR review of approach to developing statistical models designed for awarding 2020 exam results