Dear Sylvia and Jonathan
Teaching Excellence and Student Outcomes Framework
Your predecessors as President and Vice President for Academic Affairs, Professors Deborah Ashby and Guy Nason, wrote to me in March 2019 expressing their concerns about the Teaching Excellence and Student Outcomes Framework (TEF), in particular that it did not meet the standards of trustworthiness, quality and value of statistics that the public might expect.
OSR has a regulatory remit in ensuring that statistics serve the public good. It is positive that data and statistics were used to inform the TEF rating process, along with contextual information and provider submissions. We want to ensure that such use of the data meets the principles of trustworthiness, quality and value set out in the Code of Practice for Statistics. We were therefore encouraged that the Independent Review into the future of TEF led by Dame Shirley Pearce included an in-depth evaluation of the statistical elements of TEF by ONS as part of the review. We considered the ONS review to be thorough, and that it investigated and set out a range of recommendations covering the main statistical issues that you raised.
As part of the development of TEF, the UK Government considered a broad range of views about the use of data in the ratings, both technically and from a practical perspective. However, as you raise, the way that data and methods are used and the transparency and reproducibility of the awards has been subject of continued debate.
The Independent Review reported its concerns with the way that data are used in the assessment process. In particular, it recognised that improvements are needed in the management and communication of statistical uncertainty at all levels including multiple comparisons; small numbers (relating to both datasets and providers) and non-reportable metrics; and relative versus absolute comparisons. It also concluded that there are limitations in the data at subject level and significant variation in the experience and outcomes of students studying different subjects within different HE providers.
The review recommends that the overarching principles of TEF should be transparent, relevant, and robust. Transparency is a central tenet of the Code of Practice for Statistics, and fundamental to building trustworthiness, and we endorse the need for transparency of the entire TEF award process, including publishing details of the methods and data used, and how they are used in arriving at TEF awards.
In addition, we consider there are two broad areas highlighted by the Independent Review that will help address many of the concerns that you have raised as the TEF continues to be developed.
First, the independent review noted the general view that the metrics have too great an influence on the process rating and that there is an anchoring effect created by giving the metrics priority and using them to establish an initial rating. We would expect the revised framework to be able to ensure that panels are able to better take account of statistical issues such as uncertainty in the data and a lack of comparability of different institutions.
Second, the shift in emphasis of TEF from being a tool for comparing institutions to predominately one that motivates improvement is welcome. This shift should help to resolve your concerns about how quantitative data are used in the process of allocating awards, for example the use of comparative data from dissimilar institutions. The Independent Review notes that TEF will contribute to the broad base of information available to inform applicants’ choices. Documenting the strengths and limitations of TEF outcomes will be important in order to assist appropriate use, and for a wider range of information resources such as Discover Uni to be clearly highlighted for users to consider when making choices for further study.
We understand that the Office for Students (OfS) plans to consult on the design of the new framework for TEF this year. To support this, it would be to the benefit of the TEF process for OfS to continue to engage with statistical experts and informed stakeholders such as the RSS and the ONS and of course the HE providers as it develops and implements its plans. As we emphasised in our recent review of the approach to developing statistical models to award 2020 exam results, engaging with a wide range of experts is an important step to ensure public confidence. We will continue to engage with OfS during this process to press for the application of the best statistical practice possible and for an increased focus on transparency for users as well as for providers.
Director General for Regulation