OSR review of approach to developing statistical models designed for awarding 2020 exam results

The Office for Statistics Regulation (OSR) will conduct a review of the approach taken to developing statistical models designed for awarding 2020 exam results. 


Our review will consider the extent to which qualifications’ regulators across the UK developed their models in line with the principles set out in the Code of Practice for Statistics 


The review will seek to highlight learning from the challenges faced through these unprecedented circumstances. We will not review the implications of the model on individual results or take any view on the most appropriate way to award exam grades in the absence of exams. 


We will try to minimise the burden of our review on organisations involved in awarding exam grades and will contribute our findings to other relevant reviews where appropriate. 

We plan to publish the findings from our review in September.

See the latest updates on our 2020 exam results review.

Statement from the Office for Statistics Regulation on Grade adjustment in UK exams in 2020 

Due to COVID-19, exams in schools, colleges and other settings across the UK have been cancelled. This year, grades awarded are based on teacher estimated grades submitted via the awarding organisations to the qualifications’ regulators. Teachers also provided rank orders for each candidate from their centre within the defined bands. These estimated grades are known as Centre Assessment Grades (CAGs). The qualifications regulators (the Office of Qualifications and Examinations Regulation (Ofqual), Qualifications Wales, the Scottish Qualifications Authority (SQA) and the Northern Ireland Council for the Curriculum, Examinations and Assessment (CCEA)) have then moderated these results through statistical standardisation models. The Office for Statistics Regulation (OSR) has received a number of enquiries about our role in regulating the statistical models. This statement aims to clarify our position.  


The models used to determine grades awarded are complex and have been developed in a short time frame to meet the deadline for results days. The models have been the subject of debate in parliaments and the media. They have generated widespread interest from stakeholders who rightly have a strong interest given the potential impact of the results awarded on the future of individuals.  


It is not within OSR’s role to regulate the operational use of models by government and other public bodies. We would not typically undertake a review of the implications of the model for individual results awarded. Understandably this is a critical factor for many and it is important it is done in a fair and transparent way. Our focus is that statistics produced by public bodies serve the public good. As bodies that do or are soon to produce official statistics, we have a role in engaging with the qualifications regulators and expect them to use statistics in line with the principles set out in the Code of Practice for Statistics.  


We have engaged with the qualifications regulators across the UK to emphasise the importance of transparency. Where models are used we want to see details of the methods used published when results are shared, as we saw from SQA. We have supported Ofqual and the other regulators in their decisions not to publish the models in advance of the results being released. This is to ensure that it did not lead to centre-assessed grades being influenced, or multiple organisations attempting to recreate the models and produce confusing or conflicting estimated results in advance of official results being published. In determining the appropriate models we support Ofqual’s attempts to be transparent including through a consultation, convening a technical advisory group and holding a symposium. We also welcome other qualifications’ regulators efforts to engage and be transparent, for example, through public videos by CCEA and Qualifications Wales. 


We will continue to work with the qualifications’ regulators to ensure statistics they produce are in line with the requirements set out in the Code of Practice for Statistics.  

Measurement of deaths related to COVID-19

There are multiple sources of information on deaths related to COVID-19 in the public domain. There are valid reasons for this range of figures which reflect the variety of uses of these data and the various methods and assumptions underlying them. However, publication of multiple sources with different counts has the potential to cause confusion and at worst mislead. Some factors causing differences in counts are set out, but the amount each factor contributes to the differences does not appear to be fully understood and needs to be more clearly explained.

For England, there are three main sources of data on daily deaths related to COVID-19:

In all three cases the organisations make information available on when deaths have been reported or registered as well as the date the death occurred. Each organisation also publishes information on methods and definitions. However, the scale of the impact of some of the methods choices and data sources is not clear.

It is positive to see organisations trying to better understand these data. Planned analysis by ONS and PHE will look at differences between published figures. This should offer a valuable insight into what is driving the differences and whether there are any changes needed in the production or interpretation of any of these statistics.

It is critical that there is greater transparency around how estimates are produced and what is driving differences between the sources. Statisticians across relevant bodies must take a lead in understanding these data and communicating weaknesses, limitations and a coherent narrative. This will improve confidence in the data and decisions made on the basis of these data.

We have today also published a blog setting out the issues between data sources in more detail and seeking greater clarity.

Scottish Government use of COVID-19 prevalence rates

Concerns have been raised with us about Scottish Government’s quantified comparisons of the prevalence of COVID-19 in Scotland and England.

The data sources used by Scottish Government in making these comparisons are unclear and therefore we do not yet have evidence to support the validity of these comparisons.

We continue to liaise with Scottish Government to clarify the basis for the statements and will share our view publicly as soon as we can.

Our Review of NHS Test and Trace statistics in England

Today we have published our Rapid Review of NHS Test and Trace statistics (England).

The pace at which the NHS Test and Trace Programme has been developed has been unprecedented. It is clear that providing these timely statistics is the result of a huge amount of work by individuals across a range of teams and organisations.

We welcome the publication of this important data in an orderly release which demonstrates a commitment to the Code of Practice for Statistics. We also appreciate the openness with which the Department of Health and Social Care (DHSC) has engaged with us throughout our review.

The volume of data in the public domain and the wealth of questions around Coronavirus that currently remain unanswered make this a challenging environment. DHSC should be at the forefront of addressing these challenges.  DHSC should develop clearer information on how data in the Test and Trace publication fit with other statistics or research outputs as well as prioritising developments to the publication which focus on understanding the effectiveness of the NHS Test and Trace Programme.

We welcome DHSC’s commitment to improving the statistics and look forward to seeing continued developments.

COVID-19 Local Area Data

Following announcement of the first local area lockdown – where it has been widely quoted that Leicester accounted for 10 per cent of all positive cases in the country over the last week – there has been an even greater interest in the local area data on COVID-19 cases and tests.

To meet this immediate need for data in the public domain Public Health England (PHE) has published the data underlying the graphic in its weekly surveillance report. Figure 9 in the accompanying dataset provides the weekly rate of COVID-19 cases of people tested under Pillars 1 and 2 per 100,000 population by upper-tier local authority in England. The rate of cases in Leicester shown in the table (140.2 cases per 100,000 population) is more than double the rate in any other local authority area.

It is likely demand for local area data on COVID-19 will continue to increase and we understand health bodies are expecting to publish further helpful data over the coming days.

OSR will continue to work with producers of statistics on COVID-19 to make the case for improved data in the public domain. You can see more of our work in this area on our dedicated COVID-19 webpage.

Data and Statistics on COVID-19 impacts on the Care sector

The impact of COVID-19 on those in care settings – care home residents and recipients of domiciliary care – has received significant media attention. There is high demand for trustworthy, quality data and statistics to understand the large number of deaths in the care sector during the period of the pandemic.

Statistics on the care sector – including care home outbreaks, number of suspected COVID-19 cases in care homes, and registered deaths in care homes – are currently released through a variety of different reports including daily and weekly surveillance reports and within weekly registered death releases (see footnote). These statistics start to provide a picture of the impacts on those receiving care and help decision-makers to understand and manage COVID-19 within care settings. However, further analyses are needed to provide context and facilitate a better understanding of key areas of concern.

The Office for Statistics Regulation (OSR) recognises producers have been making improvements to reporting in this area. For example:

These new data and the efforts of the producer teams are welcomed, and we recognise that producers are seeking to develop statistics provision in this area. Gaining a better understanding of impacts may take some time, as fuller and more complete data becomes available, and will involve efforts from both official statistics producers and the wider research community, working together where there are beneficial collaborative opportunities. Producers should coordinate their efforts and work together to play a key role in providing data and statistics to meet user needs.

To further improve these statistics, OSR recommends that producers continue to work together to present a coherent picture of the impact on those in care settings across the UK during the pandemic. Producers need to provide users with data to develop a better understanding of the large number of deaths and the progression of COVID-19 in the care sector, going further in their analyses and combining the stories these datasets are telling us. This will help to understand how to manage outbreaks and reduce the impact of COVID-19. In particular we consider producers should:

  • Explain the wider context of COVID-19 and the large number of deaths for those in care settings. There needs to be greater understanding of the number of deaths in care settings, considering the increase in overall deaths as well as those recorded as involving COVID-19. In order to be able to unpick some of the reasons which may underlie the trends in data, there is a need for information to contextualise the data and statistics on deaths in the sector as well as to support management of COVID-19. For example, information on COVID-19 testing for those in care settings and patterns and practices in relation to hospital discharges and admissions of care home residents during the stages of the pandemic.
  • Understand and assess the impact of any changes in the circumstances and context of data sources, and any implications for use should be clearly explained. Within the varied landscape of statistics and data on those in care settings, producers should make it clear to users the definitions within their outputs – for example deaths involving COVID-19, deaths due to COVID-19, or deaths of those with a positive test result. Producers should also work closely with relevant parties to understand any differences in the processing or recording of datawhich may impact on the accuracy of statistics on deaths in the care sector.
  • Provide or enable regional comparisons where possible, providing guidance and contextual information to support the interpretation of the statistics. Producers should also enable UK comparisons where possible, providing guidance on whether the data from different countries of the UK can be compared in order to support users’ understanding and interpretation of the statistics. The similarities and differences between the country-level data should be clearly explained particularly any differences in care provision, differences in the characteristics of the population of those receiving care, and data collection methods that could affect the ability to make comparisons.

OSR will continue to monitor the data on the impact of COVID-19 on those in care settings and will work with the producer teams to support them with the recommendations outlined in this statement.

Footnote – list of official data on COVID-19 in the care sector

Public Health England – Weekly COVID-19 surveillance report – Data from a variety of different sources: community, primary care, secondary care, virology, mortality surveillance and sero-prevalence surveillance data.

Public Health England – COVID-19: number of outbreaks in care homes (management information) – Weekly number and percentage of care homes reporting a suspected or confirmed outbreak of COVID-19 to PHE by local authorities, regions and PHE centres.

Office for National Statistics – Number of deaths in care homes notified to the Care Quality Commission, England – Provisional counts of deaths in care homes caused by the coronavirus (COVID-19) by local authority.

Office for National Statistics – Deaths registered weekly in England and Wales – Provisional counts of the number of deaths registered in England and Wales, including deaths involving the coronavirus (COVID-19), by age, sex and region, in the latest weeks for which data are available.

Welsh Government –  Notifications of deaths of residents related to COVID-19 in adult care homes – Notifications to Care Inspectorate Wales of deaths of residents related to COVID-19 in adult care homes during the coronavirus pandemic.

Northern Ireland Statistics and Research Agency – Weekly death registrations in Northern Ireland – Provisional counts of the number of deaths registered in Northern Ireland, including deaths where COVID-19 was mentioned on the death certificate, by date of occurrence, age, sex, local government district, place of death and place of residence.

Department of Health Northern Ireland – COVID-19 Dashboard – Daily data on testing, cases, deaths (where there was a positive COVID-19 test) by setting, local government district, gender, age group and date; admissions by suspected and confirmed COVID-19, hospital, Health and Social Care Trust, age group and date; ICU and hospital bed occupancy by confirmed/suspected COVID-19 and date; suspected, confirmed and closed care home outbreaks by date.

Public Health Agency – Monthly Surveillance Bulletin – Data from Health Protection surveillance and notification systems (in Northern Ireland) on: laboratory confirmed COVID-19 cases; notifications of confirmed or suspected COVID-19 care home outbreaks for NI and by Health and Social Care Trust, week and cumulative percentage; primary care surveillance; critical care admissions with confirmed COVID-19; mortality (deaths (links to NISRA) and excess deaths) surveillance; and COVID-19 testing, including the proportion positive.

Scottish Government – Coronavirus (COVID-19): daily data for Scotland – Daily data on testing and suspected and confirmed cases.

Scottish Government – Coronavirus (COVID-19): trends in daily data – Past data and trend charts for the daily updates on COVID-19 in Scotland.

Scottish Government – Coronavirus (COVID-19): adult care homes – additional data – Weekly data on suspected cases of COVID-19 in adult care homes in Scotland.

National Records Scotland – Deaths involving coronavirus (COVID-19) in Scotland – Provisional statistics on the number of deaths associated with coronavirus (COVID-19) and the total number of deaths registered in Scotland.

COVID-19: Production and use of management information by government and other official bodies

This guidance note summarises our position and expectations on the use of management information by government and other official bodies

During times of rapid change there is an increased need for timely and detailed data. It is important that ministers have up-to-date information to inform government’s response to the coronavirus pandemic.  This can include such data on how to manage hospital capacity, or how to most effectively process the increased volume of benefit claimants. This type of data, used to inform operational delivery, policy decisions and measure operational performance, are often termed management information.

With increased scrutiny of all decisions, and a greater thirst for timely information from the media and the public, it is unsurprising that ministers and other public figures find themselves quoting management information in public forums, such as in parliament and in media interviews.

The Office for Statistics Regulation (OSR) recognises the importance of management information to the operations of government. There is currently a need to share information more widely than usual to inform decisions in response to the pandemic and the efforts being made by analysts to meet the increased demand for timely management information. It is clear that statisticians are focused on getting the most relevant data into the public domain, including where necessary, reprioritising outputs to allow for new publications.

Our expectations

Notwithstanding these positive developments, OSR recommends that producers and users of management information should be guided by the following three principles of use.

Equality of access: When management information is used publicly to inform Parliament, the media and the public, it should be published in an accessible form, with appropriate explanations of context and sources.

Regulatory intervention: In cases where management information is quoted and is material to public debate, we will consider a public intervention to highlight the failure to preserve equality of access.

Proportionality: We recognise there will be occasions when ministers refer briefly or in passing to the management information they have access to in responding to questions. We will be proportionate in judging when we step in, and we will distinguish between a one-off use that is marginal to the issues of interest, and those figures that are material to public debate.

Delivering on these expectations

The use of management information in the public domain

Given the volume of data flowing around government and the pace at which things are changing, there are inevitably instances when unpublished figures are being quoted in the public domain. It is right that ministers have access to up to date information. It is also right that this information is shared with the media and the public, but it remains important this is done in a way that promotes transparency and clarity. Otherwise it has the potential to cause confusion and undermine confidence in the statistics and organisations that produce them.

The Code of Practice for Statistics

While we appreciate the need for pragmatism during this unprecedented time, the Code of Practice for Statistics provides principles which should guide the release of management information as well as official statistics. It sets out clear expectations that organisations should commit to releasing statistics in an open and transparent manner that promotes public confidence – this includes being clear about appropriate caveats or quality concerns and ensuring a coherent narrative across different sources of information. It also highlights the need for organisations to look after people’s information securely and manage data in ways that are consistent with relevant legislation and serve the public good.

The role of Head of Profession for Statistics

An organisation’s Head of Profession for Statistics should play a key role in advising on the use of data and in guiding judgements. As an example, difficult choices may be needed on whether data are of sufficient quality to support the use being made; professional analysts are well placed to advise, and to balance the inherent risks.

We encourage public organisations to work with Heads of Profession for Statistics to ensure the Code standards of Trustworthiness, Quality and Value are considered as data are used to assist the government’s response to COVID-19 and support public accountability.

National Statistician’s guidance on management information

OSR endorses the National Statistician’s guidance on handling management information (October 2019). It highlights four high-level principles that should be considered:

  • Maximum value should be made of the information
  • Equality of access to data on which public statements are based
  • Transparency should guide decisions about the use and release of data
  • Integrity of official statistics – nothing should be done to undermine confidence in the independence of related official statistics


COVID-19 surveillance and registered deaths data review

Information available on COVID-19 cases and deaths has been developed rapidly in a constantly shifting environment. The work being done by analysts to get this information into the public domain is commendable. There will always be a desire for improvements to the timeliness and completeness of data, but this should not undermine the huge efforts being made by individuals and organisations to deliver timely data to support decision making and inform the public.

Our vision is statistics that serve the public good. We aim to support producers of statistics and data to achieve this while championing the needs of the public. We have undertaken a short review of the data releases on COVID-19 cases and deaths – at a UK level and for each country within the UK – to help understanding of the available sources and to highlight strengths and our view on areas for improvement. This document outlines the findings from our review, that is necessarily only a snapshot of what are very fast-moving developments.

In reviewing the various statistical outputs, we have been guided by the three pillars of the Code of Practice for Statistics: Trustworthiness, Quality and Value. Trustworthiness refers the governance that surrounds the production of statistics; Quality refers to the characteristics of the data; and Value considers the extent to which the statistics answer users’ questions.

Summary of findings

There have been many developments to the data and supporting information available on COVID-19. Analysts have made huge efforts to deliver the information and have shown a willingness to address concerns and make rapid improvements.

There is great value in having timely data, such as the daily surveillance data covering the UK that is published less than 24 hours after the data reporting period. It provides an important leading indicator of the trend in COVID-19 testing, cases and deaths, which is essential to inform operational decisions being made at pace. However, the speed at which these data are made available means there has been a trade off with completeness, and the limitations of the UK data have not been fully explained.

The nature and extent of the uncertainty around the UK estimates of deaths associated with COVID-19 has not so far been made clear. However, we are aware of efforts being made to improve the clarity and transparency of the material that accompanies the daily briefing, including drawing on support from the Government Statistical Service (GSS).

In contrast, the weekly death statistics published for England and Wales, Scotland and Northern Ireland provide a more complete measure of the number of deaths associated with COVID-19, but these statistics are released with a greater time lag.

ONS’s publication of its forward workplans in this area is a helpful development for stakeholders and it is important that other nations provide detail about their plans to keep users of the statistics informed. We understand that the GSS is considering the accessibility of all the information on COVID-19 to allow users to navigate all outputs from a central hub, such as the GSS health and care statistics landscape.

Areas for further development

  1. It is important to maintain public confidence and trustworthiness of statistics that are used to inform public debate. The nature and extent of the uncertainty around the UK estimates of deaths associated with COVID-19 should be clarified.
  2. All statistics producers should show they are actively considering the diverse and changing user need for COVID-19 statistics, by publishing detailed plans for improvements, for example, information about the occupancy of intensive care units or beds, or on person characteristics, such as ethnicity.
  3. The GSS should consider the accessibility of the information and allow users to navigate all COVID-19 related outputs from a central hub, such as the GSS landscape.

Statement from the Office for Statistics Regulation – COVID-19 Update

We welcome the work of analysts across a range of organisations in providing the public with information related to the COVID-19 pandemic. The pace at which these organisations have set up new data collection and dissemination processes has been unprecedented and enabled timely updates on the number of COVID-19 cases and deaths. We warmly congratulate all those who are contributing to this effort.

Estimates for the number of cases and deaths for the whole UK are being published and each of the four nations within the UK should continue to collaborate to enable UK reporting as statistics are developed further.

In our discussions with producers of statistics, we have seen a commitment to continuously improve the information provided to the public. Following these discussions, there have been improvements to provide greater clarity, including:

  • clarification of what the daily figures published by Department of Health and Social Care (DHSC) do and do not include (those who have died in hospitals and who have tested positive for COVID-19);
  • improvements to the supporting information on the Public Health England (PHE) dashboard, including clearer explanation of the sources and coverage of these figures;
  • explanation in the Office for National Statistics (ONS) weekly deaths statistics of how COVID-19 related deaths impact on the figures;
  • clarification of how ONS and DHSC figures relate to each other through the joint statement published by DHSC and ONS;
  • cessation of publication of a “patients recovered” figure by Public Health England because of data quality limitations.

Clear explanations of what the data mean for Northern Ireland and the commitment to regular publication times in Scotland, Wales and Northern Ireland also support user confidence.

It is important that statistics producers continue to enhance the information available to the public. We have been assured that statistics producers are working to make developments, including:

  • further breakdowns of the data and more information about hospital admissions;
  • more information on the capacity of and demands on the health system, building on the analysis of NHS 111 calls published by NHS Digital;
  • further explanation of how the figures from the UK’s four nations compare to one another;
  • and, in the medium term, greater information about the demographic characteristics of people who are confirmed as having or having had COVID-19.