Quality and clarity

17. The KS4 statistics are produced using several sources of administrative data including GCSE results and vocational qualifications from awarding bodies, demographic information from the School Census and KS2 performance data. DfE has published a wide range of information available on its methodologies and the quality checks that it carries out. Much of this information can be found on the KS4 methodology page and some of it can be found in standalone documents, for example How Progress 8 and Attainment 8 measures are calculated.  
18. DfE appears to have robust and detailed quality assurance checks in place and information on these is included in the methodology documents. Users that we spoke to stated that the quality of the statistics was very good and that they were generally happy with the information provided about the limitations, coherence and comparability. ‘Level of Quality’ scored the highest level of satisfaction in our online questionnaire, however respondents were less satisfied with ‘Advice about strengths and limitations’ which score the second lowest out of the ten aspects that we asked about.  
19. A strength of the KS4 statistics is that they are based on administrative data which offer reliable granularity and the possibility to merge the data with other administrative data held by DfE to provide detailed information on sub-groups of the population. However, there are some limitations of the data sources that are not fully explained to users such as population coverage or potential biases. Examples include the proportion of awarding bodies that provide data to DfE and any potential biases resulting from this, and information on the populations excluded from the statistics. DfE has processes in place to mitigate the impact of any limitations and we consider that those listed above do not affect the quality of the data, only that users should be able to easily find information on them.  We would encourage DfE to provide clearer information around any potential limitations to users so that they are able to draw their own reliable conclusions from the data.  
20. The statistics are released alongside an accompanying commentary as part of the statistical bulletin, which presents high-level measures with contextual information. The bulletin is clearly written with distinct sections, and the figures are presented in interactive charts, which users report are easy to interpret. Summary tables are included, which allow users to see the statistics without having to download additional tables. Users were largely positive about the commentary that accompanies the KS4 statistics and we found the bulletin to be a strong example of statistical presentation.  
21. At the top of the statistical bulletins, DfE provides information on the corrections that it has made to the statistics since publication. This includes descriptions of the specific statistics that have been corrected along with explanations of why the changes have been made. It would also be useful if more detail was included about the scale and the impact of the changes that have been made in order to maintain users’ trust in the data.   
22. Overall, users were very happy with the amount of underlying data that DfE makes available through the data downloads and through the table building tool. However, a common theme that was raised was the lack of consistency across historical data. For instance, users reported that the name of some variables in the datasets change from year to year, making it difficult to conduct a longitudinal analysis without additional data processing.  
23. DfE is aware from its own user research that the Attainment 8 and Progress 8 measures can be difficult for some users to interpret. The introduction of the new accountability measures such as Progress 8 and the discontinuation of previous accountability measures has made it difficult to compare school performance over time due to the break in the time series. Whilst DfE has published detailed information on how Progress 8 is calculated, the document contains little information on the limitations of this metric.  
24. Where users did have concerns about the quality of the KS4 data and the methodology information provided, these mostly related to the upcoming 2021/22 release. There have been a number of significant changes to the KS4 statistics over the last few years due to the effects of COVID-19 which mean that not all of the statistics can be compared over time. Users are concerned about the return of Progress 8 in 2022 and highlighted that the publication will need to be appropriately caveated.  
25. It is essential that the 2021/22 release contains comprehensive information on the strengths and limitations of the KS4 performance statistics and that it is clear which comparisons can be made. We are aware that DfE has already been undertaking user research to inform the best way to present this information.