Dear Philip,
OSR review of approach to developing statistical models designed for awarding 2020 exam results
As you are aware the Office for Statistics Regulation is undertaking a review focused on the process of developing the statistical models designed for awarding 2020 exam results. I am writing to update you on the scope of our review and our current approach.
Our review aims to identify lessons for other public bodies considering the use of statistical models to support decisions. The lessons will be based on our consideration of the extent to which the qualifications’ regulators across the UK developed their models in line with the principles set out in the Code of Practice for Statistics. To inform our review we are initially exploring the following two questions.
To what extent did the approach to developing and communicating the models support public confidence in the outputs?
We will be considering clarity of purpose, governance, transparency, communication, ethics and public acceptability of the algorithms as well as the role of the appeals process. We anticipate that compliance with the Code’s Trustworthiness principles, which support confidence in the people and organisations that produce statistics and data, will be a key focus area.
To what extent were the algorithms developed appropriate for their intended purpose?
We will be exploring the model selection and limitations, quality of the data sources, quality assurance of outputs, innovation and collaboration during the model development and validity of claims around the algorithm. We anticipate that compliance with the Code’s Quality principles, which support practices to ensure statistics are fit for their intended uses and are based on appropriate data and methods, will be a key focus area.
Our focus will be on the learning specific to the application of the Code of Practice for Statistics and will seek to highlight learning from the challenges faced through these unprecedented circumstances which can be applied across Government. We will not be reviewing the implications of the models for the results of individual students or evaluating government policy decisions on the most appropriate way to award exam grades in the absence of exams.
We are gathering evidence to inform our review via published documentation and through meetings with wider stakeholders and with key members of your team. We appreciate the co-operation shown to date from your team in setting up initial meetings with ourselves.
We are mindful of the pressures that teams and individuals are under at present and will endeavour to minimise the burden of our review on all the organisations involved in awarding exam grades. We will also look to contribute our findings to other relevant reviews where this is appropriate.
We aim to publish findings at the end of October.
I am happy to discuss this further and answer any questions that you may have. I will continue to provide updates to you as the review progresses.
I am copying this letter to Steve Davies, Director of Education at Welsh Government.
Yours sincerely
Ed Humpherson
Director General for Regulation