Dear Neil,
Transparency of trust-led school system evidence for education White Paper
I am writing about the evidence document The case for a fully trust-led system that was published alongside the education White Paper Opportunity for all: strong schools with great teachers for your child in March 2022. We have reviewed the document and have identified issues regarding the transparency, quality and replicability of the statistics presented as outlined below. We have spoken to your team responsible for producing the evidence document about the issues raised in this letter and we welcome their plans to update the document.
- Transparency: It is not always possible to identify the exact data that have been used to produce analysis in the document. Where data are referred to in the text, links to the sources of data should be clearly set out and enable users to easily find the specific data referenced. Our expectations are outlined in our transparency guidance.
- Quality: Insufficient information has been included on the methodologies used to produce the novel statistics presented in the document. In addition, the limitations of these methodologies and the implications that these would have on the fairness of the comparisons being drawn have not been fully explained to users. The Department for Education should include clear information on the methodologies and associated caveats so that users can draw reliable conclusions.
- Replicability: The limited transparency around the data sources and methods means that it is difficult for users to replicate the figures presented and to draw their own conclusions. The Department for Education should ensure that sufficient information is included in the document so that users are able to easily replicate the statistics.
We recognise that this document was produced to accompany the White Paper and is not a purely statistical document. This approach risks conflating the policy proposals with the statistical analysis presented. Care should be taken when producing these types of documents to ensure that the statistics are presented clearly and independently of the conclusions drawn from them.
Given the high-profile nature of the White Paper that this evidence document supports, the Department for Education should ensure that users have access to key information to be able to understand the data and what this means for the conclusions that can be drawn from the data. This will support trust in the data and decisions made on the basis of these data.
We understand that your team is planning on making changes to the evidence document as well as publishing an additional document which will contain the data and associated methodological information. This additional information will be valuable to users and will help inform the public debate and we are pleased to see the swift action taken by your team to make this further information available.
Yours sincerely
Ed Humpherson
Director General for Regulation
Related links
Dr Mary Bousted and Kevin Courtney to Ed Humpherson: Transparency of trust-led school system evidence for education White Paper (1 April 2022)