Dear Professor Jerrim,
Thank you for your letter of 22 April and your paper raising your concerns around the way the quality of data from the 2018 Programme of International Student Assessment (PISA) had been reported. As you highlight the PISA data are widely used to evaluate the education systems across the UK. It is important that the quality of the data for this use is clearly explained. You asked us to carry out an independent review of the PISA data across the UK.
My team has held constructive conversations with the relevant statisticians in the four nations of the UK. We have reviewed the reports against the Code of Practice for Statistics and have written to the lead statisticians with our findings and recommendations.
The key findings of our review against the Code are set out as follows.
Quality information
The technical standards set out by the OECD ensure a high level of quality in the results. The national reports make it clear whether differences are statistically significant or not. However, we agree with you that the reports were not sufficiently transparent about the limitations of the data and the potential sources of bias. Each of the producer teams recognises that reporting information about the quality of the statistics is important for users to understand the impact of limitations and sources of bias on their use of these data, and each has committed to publish more information about quality for future rounds of PISA. This should include information about potential limitations of the data and potential sources of bias. Where a non-response bias study is required, the results of it should be published in full. In addition, any changes to the PISA data collection, including to the timing, should be clearly explained along with any potential impacts on the results.
Application of the Code
Whilst the report in Scotland is an official statistics product, and therefore Scottish Government sought to be compliant with the Code, the reports in the other nations are research reports. Whether published formally as official statistics or not, we encourage statistical producer bodies to comply as fully with the principles of the Code of Practice for Statistics as they are able. We know that the respective statisticians are committed to upholding the highest standards of Trustworthiness, Quality and Value for their outputs. Therefore, we encourage all four nations to comply with the Code as fully as possible.
User engagement
The statistics teams in all four nations engage extensively with the relevant policy teams within their organisations to ensure that their needs are met. There are also good examples of wider engagement, such as the briefing provided to the media by Scottish Government and inclusion of academics on its advisory panel. However, the producer teams should seek, and respond to, the input of users beyond policy to ensure that public value from PISA data is maximised.
We trust that these recommended actions will help ensure that your concerns are addressed for future PISA reports. If you would like to discuss any of these issues in greater depth, my office would be happy to arrange a meeting.
Yours sincerely,
Ed Humpherson
Director General for Regulation