Dear Roger

Reporting of data from the Programme for International Student Assessment (PISA) in Scotland

I am writing to you following our recent review against the Code of Practice for Statistics of your publication Programme for International Student Assessment (PISA) 2018: Highlights from Scotland’s results.

We would like to thank your team for their cooperation in helping us with our review. It is good that since the publication of the 2018 results, your team used a self-assessment process with an internal group of analysts to review the report and identify areas where future reports could be strengthened, particularly in terms of the reporting of the quality of data.

Your team also indicated that following user interest, it plans to work with the University of Edinburgh to link the data to National 4 and National 5 exam results. We are pleased that user requirements such as this are being addressed and encourage any future initiatives that can yield further public value from the data. Furthermore, it was encouraging to hear about some of the outreach work that your team is doing and the desire to impart knowledge learned from the data to help academic users use the data more effectively.

Our review found that the production of the report within Scottish Government as official statistics meant that compliance with many aspects of the Code of Practice was apparent. These included the publication of a range of material about the quality of the statistics, pre-announcement of the publication and a short list of people granted privileged pre-release access to the statistics. The fact that the statistics produced by your team are extracted directly from open data available publicly from the OECD PISA website also ensures transparency and enhances trustworthiness.

As we discussed when we met with your team, we have identified some ways in which you may be able to improve the information provided about the quality of the statistics. I welcome your commitment to progress these points:

• Even though Scotland exceeded the OECD minimum threshold of 80% pupil participation rate and was not required to carry out a non-response bias analysis (NRBA). As the participation rate was just over the minimum threshold there may be merit in doing so and reporting the results, to help users better understand quality and to provide reassurances to users about bias.
• Better explanation about why pupils’ withdrawal rates had increased and more transparency around ineligibility of students would increase the robustness of the narrative about methodology and choices made on sampling. Further guidance around communicating change is available from the Government Statistical Service Communicating Quality, Uncertainty and Change.
• More clarity around the effect on PISA results of the change in the reporting period, which resulted in a greater proportion of pupils taking part in 2018 being from Secondary 5[1] (S5) compared with previous years would help users better understand how they might use the data.

We would be happy to talk to your teams in more detail on any of the findings above.

Yours sincerely

 

Mark Pont

Assessment Programme Lead

[1]  Secondary 5 (S5) is the Scottish equivalent of Year 12 in England and Wales. Pupils normally sit National 4 and National 5 exams in Secondary 3 and 4 and Scottish Highers in Secondary 5.