Requirement:

Improve the distinction between the KS4 accountability measures and the wider KS4 statistics by investigating and implementing changes to the presentation of the statistics in the statistical bulletin.

Actions taken and commitments made by DfE to meet the requirements:

DfE made a number of changes to the 2021/22 publication in order to improve the distinction between the KS4 measures. These included:

• Updating the structure of the publication to more clearly divide the different KS4 measures.
• Adding additional information to the ‘Which KS4 headline measures are being reported on?’ section including wording such as “The KS4 measures are designed by the Department to encourage schools to offer a broad and balanced curriculum with a focus on an academic core.”
• Adding more information, including worked examples, into the secondary accountability measures document to help users understand these measures.

DfE noted that that this was the first full publication since the Covid-19 pandemic and the first time publishing the full suite of statistics in October (as opposed to January). As such, the scope for large changes was limited at this time but DfE has stated that this is an area that it will continue to look into to identify possible improvements.

OSR’s evaluation of evidence:

We welcome DfE’s changes to the publication including to the structure and the level of information provided on the accountability measures. These changes will help users to more easily interpret these measures as well as understand the purpose of them in the political context.

We support DfE’s plans to investigate further improvements that could be made in this area.

Requirement:

Ensure users are appropriately informed about the impact of the COVID-19 pandemic on the upcoming 2021/22 release and that they are able to draw their own reliable conclusions from the KS4 performance data by providing comprehensive information on the strengths and the limitations of the statistics and the appropriate comparisons that can be drawn.

Actions taken and commitments made by DfE to meet the requirements:

DfE introduced a number of changes to the different KS4 statistical products. These are summarised below.

Find school performance website

DfE has a very clear section on the find school performance website to warn users about making direct comparisons and that contextual factors should be taken into account. There is also a ‘read more’ section in this box that contains additional information for expert users. DfE is in the process of considering how the advice will be adapted for future years.

DfE is also reviewing the way that confidence intervals are presented on the school level website so that the context of the scores can be better understood by non-expert users.

DfE changed the name of the school-level website from ‘compare school performance’ to ‘find school performance’.

Statistical bulletin

In the bulletin, DfE has added in information on Covid-19 into several sections including the introduction, headline facts and figures, about this release, and a new section on comparing KS4 measures over time.

Changes to the exams are clearly marked on charts (for examples see figure 7 in attainment by disadvantage status).

Communications

DfE released several communications in the lead up to the KS4 release and on the day of the release to provide advice and guidance on how to interpret the figures:

• GCSE and Key Stage 4 vocational qualifications data explained – The Education Hub (blog.gov.uk)
• How to understand school performance – everything you need to know – The Education Hub (blog.gov.uk)
• GCSE and VTQ results – what do the stats show? – The Education Hub (blog.gov.uk)
• We’re changing the way we present school and college results data this year – what you need to know about performance tables – The Education Hub (blog.gov.uk)

DfE’s communications team also proactively engaged with the media in the lead up to the publication to ensure that the figures would be used appropriately.

OSR’s evaluation of evidence:

We consider that DfE made a number of informative changes to the find school performance website and to the statistical bulletin that demonstrated best practice and supported the appropriate use of the statistics.

The information released in the blogs in the lead up to the publication was extremely beneficial to users and it was positive that DfE engaged so proactively with the media.

It was very positive that the publication itself was well-received and provided important insights, for example into the attainment gap.

Requirement:

Ensure users are at the centre of the statistical production process by engaging effectively with the widest possible audience through sharing more information publicly on the development plans for the KS4 performance statistics and by working closely with users to meet their needs.

Actions taken and commitments made by DfE to meet the requirements:

DfE has been engaging more closely with stakeholders to discuss changes with them.

The statistics team has continued to work closely with the DfE website team to gather and implement feedback from users on how the statistics are presented. The team made changes to the recent publication to make it more accessible for machine readers and made improvements to the table building service.

DfE is in the process of reviewing its data requests and freedom of information requests to determine whether it can publish additional data that are requested frequently.

DfE is looking into ways to publicly share information on developments and changes earlier as well as publish a work plan.

OSR’s evaluation of evidence:

Whilst we judged that DfE had strong relationships with its key users, we considered that more could be done to engage with a wider audience as well as share more information publicly.

We have seen some positive progress towards this requirement, including greater engagement with stakeholders, user feedback from the website, and analysing data requests and FOIs.

We understand that DfE has plans to continue work in this space including assessing whether there is additional data that can be released regularly and publishing a work plan, through which information on upcoming developments can be shared with the public. We consider that this continued programme of work will ensure that the KS4 statistics meet the needs of the greatest number of users and that users are able to input into future developments.

Requirement:

Ensure public confidence in its data governance procedures, by publishing an update on progress with meeting the recommendations of the ICO audit that relate to the Code of Practice for Statistics and providing assurance that all data governance requirements are being adhered to.

Actions taken and commitments made by DfE to meet the requirements:

The ICO audit has now been closed. The ICO has approved actions, planned or implemented, which improves process relating to the vast majority of audit recommendations.

OSR’s evaluation of evidence:

It is very positive that the ICO audit has now been closed. We consider that a public announcement of the closure would help to build greater trust in DfE’s data governance and data protection procedures.