Dear Neil, Stephanie and Pauline
Reporting of data from the Programme for International Student Assessment (PISA) in England, Wales and Northern Ireland
I am writing to you following our recent review against the Code of Practice for Statistics of the three sets of outputs of results from the Programme for International Student Assessment (PISA) for 2018:
- PISA 2018: National Report for England
- Achievement of 15-year olds in Wales: PISA 2018 National Report
- PISA 2018 Achievement of 15-year olds in Northern Ireland
We would like to thank your teams for engaging with us effectively during the review. In reviewing the PISA 2018 data reporting, we focused on transparency and the quality of the reporting of the data. Our review found a range of areas of good practice:
- the contracting out of the national reports for England, Wales and Northern Ireland to an independent research organisation ensures sufficient technical expertise and resources to produce the national reports on behalf of the three nations
- publishing the national reports on the same day as the international data and report by the OECD
- working within an international study framework set by the OECD for all participating nations and the resultant technical standards that must be adhered to
- there are strong links between the statistical teams and users within their departments’ policy teams that use PISA data
- there are good working relationships between all four nations of the UK, and they share best practice
- the reports themselves describe the main messages and make it clear whether differences are statistically significant or not, providing users with insight to key findings
As we discussed when we met with your teams, to enhance the value of these data to users, we have identified three key areas for improvement. I welcome that each of you has committed to better comply with the Code in those areas:
- providing the relevant information about quality to help users to understand limitations and sources of bias. For example, for the 2018 data, a Non-Response Bias Analysis (NRBA) was carried out for England and for Northern Ireland. The results of these were not published and doing so would have provided reassurances to users about bias. We would encourage each of you to publish information such as any non-response bias analysis in full. There is a range of guidance currently available to help improve communication around quality, for example the Government Statistical Service’s guidance on Communicating Quality, Uncertainty and Change
- carrying out wider user engagement to help ensure that public value from these data is maximised
- considering voluntary application of the Code of Practice for Statistics, and publishing a statement explaining this to users. While we appreciate that many steps have been taken in the cycle for the next publication, complying with the Code will still be able to offer users reassurances around trustworthiness, quality and value of the statistics
We would be happy to talk to your teams in more detail on any of the findings above.
Yours sincerely
Mark Pont
Assessment Programme Lead