UK Statistical System UK Statistics Authority Code of Practice
Compliance check

Ed Humpherson to Jonathan Waller: Assessment of the Higher Education Graduate Outcomes Data and Statistics

Dear Jonathan

Assessment of the Higher Education Graduate Outcomes Data and Statistics

We have completed our assessment of the Higher Education Graduate Outcomes Data and Statistics produced by Jisc, under the Higher Education Statistics Agency (HESA) brand. I am grateful for the positive engagement from your team throughout the assessment process.

Since their introduction in June 2020 for the 2017/18 academic year, these statistics have continued to be an invaluable source of information on the outcomes and destinations of graduates from Higher Education (HE) in the UK. These statistics are a vital source of information on graduate outcomes in the UK and are used by a wide range of stakeholders including higher education providers and regulatory bodies across the UK.

Jisc has shown a commitment to trustworthiness through its understanding of its responsibilities in important areas such as data security, professional capability, and intelligent transparency. By fully maximising the positive relationships with its key stakeholders, Jisc has ensured that the needs of users are embedded in the production and continued development of these statistics.

The quality of these statistics is very high. The survey is well designed to meet users’ needs and achieves good response rates at a national level. We commend the transparency around the quality of these statistics and that a large amount of information is provided in the accompanying user guide.

Jisc’s engagement with its statutory customers and other key users through the provider forum, steering group and other means, demonstrates a commitment to strong user engagement practices. We welcome Jisc’s plans for process modernisation including the implementation of Reproducible Analytical Pipelines (RAP) as well as the Graduate Outcomes Data being made available for future research, including data linkage, on the Office for National Statistics’ Secure Research Service.

Our assessment found that these statistics adhere to the Code of Practice closely. Through the range of actions that Jisc has taken to develop these statistics, and in its response to our assessment, Jisc has shown a high level of dedication to the principles of trustworthiness, quality and value.

We judge that the Graduate Outcomes Data and Statistics can be confirmed as accredited official statistics (called National Statistics in the Statistics and Registration Service Act 2007). We would like to acknowledge that it is rare for OSR to accredit official statistics without first providing producers with requirements and that this reflects our high regard of these statistics as well as Jisc’s approach to continually maintaining the high standards of the Code of Practice for Statistics.

Yours sincerely

Ed Humpherson

 

Related Links:

Assessment of compliance with the Code of Practice for Statistics: Higher Education Graduate Outcomes Data and Statistics

Jonathan Waller to Ed Humpherson: Higher Education Graduate Outcomes Statistics Assessment Request – October 2023

Ed Humpherson to Jonathan Waller: Higher Education Graduate Outcomes Statistics Assessment Request – October 2023

Ed Humpherson to Chris Roebuck: Request for assessment of NHS England Cancer Waiting Times Statistics

Dear Chris

Request for assessment of NHS England Cancer Waiting Times statistics

Thank you for your letter of 1 March requesting an assessment of NHS England’s Cancer Waiting Times statistics.

We note the changes that you have made to these statistics in response to programme developments and user feedback. Given the high-profile nature of the statistics, I welcome your aspiration to publicly demonstrate your compliance with the Code of Practice for Statistics.

I agree with your suggestion that the assessment should commence no earlier than Autumn 2024 to allow time for the statistics to bed in. With this in mind, I am pleased to confirm that my team will be in touch with you shortly to confirm timing for this assessment.

Yours sincerely

Ed Humpherson

Mark Svenson and Chris Roebuck to Ed Humpherson: Request for assessment of NHS England Cancer Waiting Times Statistics

Dear Ed

Request for assessment of Cancer Wait Times statistics

Following the introduction of the new NHS waiting times cancer standards from 01 October 2023, we have made a number of changes to our cancer wait times statistics. The changes to the cancer waiting times publication are given on the cancer waiting times statistics publication page and were prominently mentioned in the first statistical release following the announcement. We have continued to give details of the changes in the statistical releases following the changes in the December publication.

Our development of the statistics was informed by programme developments and feedback received from the wider user community. The recent consultation had widespread support for the new completed standard measures. Feedback was received requesting further disaggregation by tumour type, and this is part of our changes to the statistics. No feedback was received requesting information to be published on the stock of patients waiting more than 104 days for treatment, therefore, balancing resource with user needs we do not plan to publish this information. We publish the distribution of the waited times for the 62-day combined completed standard, and this includes those who have waited more than 104 days for a treatment.

In accordance with Section 12(1) of the Statistics and Registration Act 2007, we would welcome a full assessment of these statistics given their continued high profile. We feel that the timing of this would be best once the statistics have bedded in. We anticipate this would be Autumn 2024 at the earliest.

We are copying this letter to our statistical governance team who are ready to pick up the next steps with the OSR team.

Yours sincerely,

Mark Svenson and Chris Roebuck

Mark Pont to Jonathan Russell: Compliance Check of Valuation Office Agency (VOA) Council Tax statistics

Dear Jonathan

Compliance Check of Valuation Office Agency (VOA) Council Tax statistics

We have recently completed a compliance check of your council tax statistics publications, Stock of Properties, 2023 and Challenges and Changes in England and Wales, March 2023 against the Code of Practice for Statistics. I hope that our reflections are helpful as you continue to develop these important official statistics. Some of the findings may also be more widely applicable to your non-domestic rating statistics, and we encourage your team to consider that broader applicability. If you would find it helpful for us to specifically review those, please do get in touch.

Your council tax publications are valuable to users who are trying to understand how council tax affects them. Breakdowns by house characteristics and number of bedrooms in the Stock of Properties release help users compare their banding with others. The Challenges and Changes publication indicates to users how likely a banding review is to increase, decrease or not change a property’s banding.

Our review found a range of positive features that demonstrate the trustworthiness, quality, and value of the statistics:

  • It is commendable that VOA has made its data available to other government departments. The use of council tax property attribute data by the Office for National Statistics (ONS) to supplement census data, and the Department for Levelling Up (DLUHC) use of the valuation list for its council taxbase statistics highlight the impact of VOA’s positive approach to data sharing and the additional statistical insights and increased efficiencies that this can lead to. For example, sharing VOA property attribute data with ONS has meant that a previous question on the number of rooms in a dwelling could be removed from the 2021 Census. VOA property attribute data is also a key source used to calculate the UK House Price Index.
  • The statistics provide granular and clear insights. They include helpful contextual information which allows users to correctly interpret the statistics and their structure is easy to follow. There is a good level of commentary and background notes to help users understand how council tax works, which will aid users in using the statistics appropriately. For example, the helpful flowchart showing the stages of the banding appeals process in the Challenges and Changes background notes outlines each stage from proposal to final decision.
  • The explanations of the methodology for the statistics should be commended, for example it is explained that the statistics are produced using ONS’s statistical geographies and National Statistics Postcode Look-up, which will help users understand the comparability of the statistics.
  • You have demonstrated a commitment to trustworthiness and transparency by making users aware of data limitations. This includes noting a key limitation, that the attributes of properties are updated only when they are sold in the background notes. The inclusion of a revisions policy and quality policy and details around VOA’s statutory duty in the background information, also show a commitment to transparency. Your team told us about your plans to publish more information on the extent of revisions made in the statistics. This should reduce the likelihood of misinterpretation and misuse, especially in the advent of legislative changes such as the planned revaluation of council tax banding in Wales, which you told us may lead to improved data quality and potential revisions to the statistics.
  • The implementation of Reproducible Analytical Pipeline (RAP) principles into the production of the statistics to improve the efficiency and reliability of the process is commendable and demonstrates a commitment to continuous improvement.
  • Your plans to further develop the statistical outputs, for example by introducing a flows measure between domestic and non-domestic tax, will give users a better understanding of the stock of taxable buildings. The potential introduction of dashboards and maps is also exciting and will be valuable to users through new ways to interact with your statistics.

To further enhance the trustworthiness, quality and value of these statistics, we have identified ways the statistics and their presentation could be improved:

  • Within the background documentation for each release you welcome feedback from users via email. However, there is little published information about your overall approach to user engagement. Having a clear user engagement strategy, especially in the context of the future planned development of these statistics, will allow users to benefit from knowing what changes are coming and facilitate an ongoing dialogue with your users.
  • The background documents for each publication are clear and well-structured but some of the information is quite general. Including specific details about approaches to data collection and assuring data quality for each set of statistics would reassure users. Publishing information, for example, around potential uncertainty in the local authority and house attribute data captured as part of a property valuation, will facilitate the appropriate interpretation and use of the data by users. This is important given the wider use of this data by other government departments. More published information, in the form of a data journey for example, could give an indication as to where uncertainty in the data may arise.
  • It was good to hear that you have previously used Quality Assurance of Administrative Data (QAAD) principles when exploring new data sources, however, we would encourage you to communicate this work to users as part of your documentation about quality. This will help users understand the quality of the source administrative data and how it can be used appropriately. A thorough appraisal of data quality will be especially important in the context of changes to data sources such as the planned changes to VOA administrative systems and the revaluation of Welsh council tax bandings. Given the extensive use of the VOA council tax and property attribute data across government, we encourage you to work with these stakeholders while you undertake future reviews of administrative data quality. This will allow you to understand data quality in relation to different uses, share resource and tailor your efforts accordingly.
  • Due to the difference in council tax policy between England, Wales, and Scotland it is important that the coherence and comparability of council tax statistics is made clear to avoid misleading comparisons. Stating which statistics can be compared and including links to relevant statistics is a good step to reducing the likelihood of misuse. The appropriate comparison of statistics is especially important with an approaching general election.
  • We welcome your commitment to reduce the number of individuals on your pre-release access (PRA) lists for both publications. This shows your commitment to trustworthiness and will reduce the likelihood that the statistics are shared more widely than is essential ahead of publication. However, your PRA lists for each publication are long, and to support ensuring proportionate access we recommend that the PRA lists are reviewed each time the statistics are published.
  • The cover sheets and notes provided in the Microsoft Excel data spreadsheets provide a good amount of information that will help users use the statistics appropriately. We recommend that similar notes are included with the CSV files as just providing raw data could lead to misinterpretation and/or misrepresentation.

I would like to thank your team for their positive engagement during this review. Please get in touch if you would like to discuss any aspects of this letter or if we can be of further assistance.

I am copying this letter to Tetyana Mykhaylyk, Director of Information, Data and Analysis, VOA; Colin Yeend, Head of Research & Analysis, Information, Data and Analysis Directorate, VOA; Sarah Windass and Anna McReady, the responsible analysts; and Sean Whellams, Head of Profession for Statistics, HM Revenue and Customs.

Yours sincerely

Mark Pont

Ed Humpherson to Ian Lonsdale: Assessment of Statistics from the People and Nature Survey England

Dear Ian

Assessment of statistics from the People and Nature Survey for England

Following our assessment of statistics from the People and Nature Survey for England produced by Natural England, we set out four requirements that we judged that you needed to meet in order for these statistics to become accredited official statistics. Accredited official statistics are called National Statistics in the Statistics and Registration Service Act 2007.

We have independently reviewed the actions that your team has taken to address these requirements. On behalf of the Board of the UK Statistics Authority, I am pleased to confirm that they comply with the standards of trustworthiness, quality and value in the Code of Practice for Statistics and should be labelled accredited official statistics.

We have been impressed with your team’s engagement and willingness to address these requirements.

I am copying this letter to Lindsey Clothier, Deputy Head of Profession for Statistics at DEFRA, and to the team at Natural England – Tom Marshall, Caitlin Clark, and Sergio Milheiras.

Yours sincerely

Ed Humpherson
Director General for Regulation

 

Related Links:

Annex: Review of actions taken in response to Assessment Report 374: Statistics from the People and Nature Survey for England (PaNS), produced by Natural England

Ed Humpherson to Ian Lonsdale: Assessment of statistics from the People and Nature Survey for England

Assessment of compliance with the Code of Practice for Statistics: Statistics from the People and Nature Survey for England

Mark Pont to Sian Rasdale: Official Development Assistance Statistics

Dear Sian

Official Development Assistance Statistics

We recently completed our compliance check of FCDO’s Official Development Assistance (ODA) Statistics against the Code of Practice for Statistics. These are important statistics that measure aid flows by UK official agencies or their executive agencies (including but not limited to FCDO) to developing countries and multilateral organisations in line with the international definition of ODA. They include total UK spend on ODA, and as a proportion of Gross National Income (GNI) in the form of the ODA:GNI ratio. These statistics have been of significant public interest not least since 2020, when the UK government announced a temporary reduction in its target spend for ODA from 0.7 per cent of GNI to 0.5 per cent of GNI effective from 2021. More recently the statistics were referred to in the media with reporting of the FCDO’s ODA programme data for its international programme for the next two years.

Our review found that the statistics demonstrate widespread compliance with the Code of Practice and should continue to be designated as National Statistics. It is clear both from the published outputs and from talking to your team how much hard work has gone into the development of these statistics and how much ambition there is to further improve them.

We found a range of positive features, as well as some areas for improvement that we consider would enhance the trustworthiness, quality and value of the statistics:

  • We commend your team for its responsiveness to user feedback by developing a dashboard, which aims to provide users with much greater insight. The dashboard when finalised, will enable users to interact with the data will bring greater transparency of UK aid data.
  • It is also good that your team considered the public value that Gross Public Expenditure statistics provided, in response to user feedback which found these statistics no longer remained relevant or useful, discontinued their production.
  • The merger of the former Foreign and Commonwealth Office and the Department for International Development resulted in the need to bring together two separate reporting accounting systems and different methods used to estimate ODA-eligible administrative costs. After reviewing these methods, against the OECD Development Assistance Committee (DAC) directives, FCDO has introduced coherence in the reporting accounting systems and an interim method to calculate the ODA-eligible proportion of FCDO administration costs. We commend FCDO on the transparency of its interim methodology.
  • In 2018 OECD DAC introduced changes to the methodology used to calculate ODA, which affected the treatment of loans. To help the public understanding and address any misconceptions of what ODA includes, FCDO published clear technical notes to explain how ODA is calculated. However, to further improve the transparency, FCDO might publish additional information explaining to users where the OECD guidance can be applied directly and where it needs to be interpreted in the UK’s local context.
  • Using a new centralised financial information database, FCDO has recently updated information on data sources, showing the wide range of other government department administrative sources used to compile the ODA statistics. The new database will also bring further opportunities to improve data collection processes. To reassure users about the quality of these sources and improved data collection processes, FCDO should consider updating its Quality Assurance of Administrative Data (QAAD) using OSR’s QAAD toolkit to support this process.

Looking beyond Statistics on International Development:

  • In October 2022 the Independent Commission for Aid Impact (ICAI) raised four recommendations in its Transparency in UK Aid review, which included:
    1. Setting and applying standards for transparency to all aid portfolios (including arm’s-length bodies)
    2. Committing to achieving a standard of ‘very good’ in the Aid Transparency Index by 2024
    3. Resuming the publication of forward aid spending plans, cross departmental development results and country aid priorities
    4. Working with other donors to support greater use of International Aid Transparency Initiative (IATI) data.
  • In December 2022 FCDO published a response to the review, where it has:
    1. Rolled out of a single finance and HR IT system throughout the department, which has allowed all ODA programmes in the FCDO to be brought onto a single platform. This has allowed for a more unified transparency process, with systems to assess, approve and collate ODA programme data.
    2. Committed to achieving a standard of ‘very good’ in the Aid Transparency Index by 2024 and publish forward looking ODA allocations and is carrying out improvements to publications of country development strategies.
    3. Committed to publishing annual reports and accounts information that support accountability and describe programme allocations clearly.
    4. Committed to supporting greater use of IATI data across all recipient countries of ODA.
  • To further improve transparency, it would be helpful if FCDO published its plans on how it will address the publication of arm’s-length bodies information, along with a record of any significant decisions to continue, discontinue, adapt, or provide statistics through other means. These decisions should be supported and driven by user feedback.

We would like to thank you and your team for your positive engagement during this review. In order to continue complying with the Code, we ask that you report back to us with your plans for addressing our recommendations above by December 2023. Our Economy, Business and Trade domain will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

I am copying this letter to Jane Casey, Head of ODA delivery, eligibility and reporting.

Yours sincerely

 

Mark Pont

Ed Humpherson to Scott Heald: Assessment of Accident and Emergency (A&E) Activity Statistics in Scotland

Dear Scott

Assessment of Accident and Emergency (A&E) Activity Statistics in Scotland

We have completed our assessment of the weekly A&E activity statistics produced by Public Health Scotland (PHS). We judge that these statistics can be designated as National Statistics once we have confirmed that the requirements set out in our report have been met. We have also carried out a compliance check of the monthly waiting times statistics and I am pleased to confirm that these statistics should continue to be designated as National Statistics. The weekly and monthly statistics are now available together on a new platform reporting statistics on A&E departments across Scotland. This platform is a positive and innovative development and welcomed by many users in terms of improved accessibility and functionality.

Waiting times continue to be of high public interest across the UK and I am grateful for the positive engagement from you and colleagues in the A&E statistics team throughout the assessment process. It was particularly encouraging how the team adopted a proactive approach and implemented many of our emerging recommendations during the development of the new platform. Of key importance is your ongoing work to ensure that users are aware of the differences in hospital site inclusions between the weekly and monthly statistics to support appropriate use of the statistics – for example, only the monthly statistics include all units and departments relevant to the Scottish Government target that 95% of people attending A&E should be seen, admitted, discharged or transferred within four hours of arrival. You publish some of the disaggregated datasets only monthly. This should be kept under review in case there is further user demand for that level of granularity on a more frequent basis.

We identified three requirements that would strengthen both sets of statistics: extending user engagement to a wider range of users, assisting users to make appropriate cross-UK comparisons, and consider ways to communicate uncertainty to further aid user interpretation of the statistics. Implementing these requirements will ensure that the statistics continue to meet the highest standards of the Code of Practice for Statistics. As discussed, we expect you to report back to us by September 2023 on how you have met these requirements.

I am copying this letter to Emma McNair, Information Consultant at Public Health Scotland.

Yours sincerely

Ed Humpherson
Director General for Regulation

 

Related Links:

Assessment of Accident and Emergency (A&E) Activity Statistics in Scotland

Scott Heald to Ed Humpherson: Assessment of Accident and Emergency (A&E) Activity Statistics in Scotland

Mark Pont to Steve Ellerd-Elliott: Benefit Sanctions statistics

Dear Steve

Benefit Sanctions statistics

We recently completed our compliance check of your Benefit Sanctions statistics against the Code of Practice for Statistics. Even though these statistics are still in development, they are important official statistics valued by users across government and those with an interest in the Benefits system. We have made several recommendations to support the continued development of these statistics.

We considered the Trustworthiness, Quality and Value of these statistics in relation to the Code and have appreciated the positive and constructive way that the team has engaged with us during our review.

Our review found many examples of good practice. We would particularly like to recognise the complexities around the development of these statistics and the re-instating of the currently suspended measures of legacy benefits sanctions, and the need to balance the pace of progress with making sure that the statistics are suitably robust.

The background information and guidance for the statistics is comprehensive and provides transparency around the issues with the data and methods and that those issues are being addressed. There are appropriate health warnings throughout the bulletin about making comparisons across benefits due to the different data sources used and methodological approaches. Making the data available for wider re-use through Stat-Xplore enables users to carry out their own further analysis and enhances the usefulness of the data.

From our conversations with your team, it is clear that a great deal of work is going on to re-instate the currently suspended measures and develop the published series further to add more value and meet user needs. This is while dealing with the limitations associated with the data being collected for operational reasons and not specifically for statistical production – for example, the data being held across several different systems and difficulties with accessing the data in a useable format. It was good to hear that regular conversations are happening between the statistics team and policy colleagues within DWP in particular to understand the context behind the year-on-year increases in reasons for sanctions decisions – an area flagged by users who would like to understand more around these reasons. The statistics team also told us that there has been a positive change in the way ethnicity data are collected during benefit applications which should enable ethnicity breakdowns to be published in future – which would address a data gap we have also seen identified by users.

However, it is difficult for users to see and appreciate this work happening as this has not been communicated to them. Sharing details of both activity to date and planned work with users would help them to understand the amount of effort that is going into making the Benefit Sanctions statistics as useful and useable as possible. When sharing your plans, you should also provide details of how you plan to prioritise work going forward so it is clear to users when developments may occur, which will also help manage their expectations.

As the plans to reinstate the suspended measures progress and with other valuable developments in the pipeline, we fully support the team’s plans to engage more widely with users beyond the regular user forums and stakeholder engagement that are already in place, and which are providing helpful channels for known users to provide feedback. Our user engagement guidance provides a useful framework to help plan these activities and this wider engagement can help inform any future decisions regarding these statistics that will help to ensure that they remain useful, relevant and continue to support understanding of this important area. The Government Analysis Function is also developing a User Support and Engagement Resource (USER) hub which contains helpful guidance on engaging with users.

In our review we identified a gap in the available quality assurance information, and we are pleased to hear from the team that it is following our Quality Assurance of Administrative Data (QAAD) guidance in order to provide both the team and users the necessary assurance about quality and to help users understand how the Benefit Sanctions data are collected and processed. The team has told us that it will publish this as soon as it is ready.

Our Labour Market and Welfare domain lead will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.

Yours sincerely

Mark Pont

Assessment Programme Lead

Mark Pont to Lucy Vickers: Adult Substance Misuse Treatment Statistics for England

We recently completed our compliance check of the Department for Health and Social Care’s (DHSC) Adult Substance Misuse Treatment statistics against the Code of Practice for Statistics. The statistics are published in a provisional monthly release supplemented with an annual report the following year. The statistics were designated as National Statistics in 2015 and provide valuable insight into adults receiving treatment for substance misuse and the characteristics of different substance users. I am pleased to confirm that these statistics should continue to be designated as National Statistics. 

We carried out this review because the responsibility for the production and publication of the statistics was transferred from Public Health England (PHE) to the Office for Health Improvement and Disparities (OHID) within DHSC in October 2021. 

Our compliance check found many positive features that demonstrate the trustworthiness, quality and value of the statistics. For example: 

  • These statistics are important for policy makers and service providers, at the local level and for England as a whole. The value of the statistics is enhanced by the commentary and context provided throughout the annual report – for example, a section is included in the statistics about the policy context, there is discussion of the impact of the coronavirus pandemic on the statistics and trends in characteristics since 2005/06 are presented. Furthermore, the statistics are published in a variety of formats to suit different users’ needs such as charts, data tables and the interactive ViewIt tool. 
  • The value of the statistics is also enhanced by the level of granularity of the different regions. The provisional monthly release provides a breakdown of trends at a local authority level, allowing users to understand their local area. 
  • It is great to hear that the team has a good understanding of users of its provisional monthly data. These users include treatment providers, clinicians, local authorities and the general public. It is also good to hear that the team consults with users every few years on planned changes to the underlying dataset. Engagement is an essential part of maintaining the relevance and usefulness of the statistics for users. 

We identified some areas that would further enhance the trustworthiness, quality, and value of these statistics, some of which the team was already considering: 

  • It is good to hear of the team’s plans to update the Quality and Methodology report published on the National Drug Treatment Monitoring System this year. The report is exceptionally detailed and provides good assurance to users regarding the quality of statistics. However, the report was last updated in 2020 so it is not clear whether there have been changes to quality or methodology that users should be aware of. We consider it would also help users’ understanding if you were to add a short summary of the essential aspects of the statistics’ quality and methods, such as any limitations or what is and is not included in the statistics, within the annual report. This would help support appropriate use of the statistics without having to access a different website.  
  • While the team has a good understanding of users of its monthly statistics, we support its plans to expand user engagement activities relating to the annual report. Gaining a better understanding of what users require from the annual publication over and above the monthly statistics and what users want regarding the presentation of the statistics would be helpful as you continue to develop them. 
  • Finally, it is good that the team has committed to make some small changes that will enhance the demonstration of the trustworthiness of the statistics. This includes adding contact details to the annual release, adding information about roles of individuals to the pre-release access list and providing greater clarity regarding the move from PHE to OHID.   

I would like to thank your team for their positive engagement with us during this review. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter.  

Yours sincerely 

Mark Pont 

Assessment Programme Lead  

Mark Pont to Sarah Crofts: Compliance review of ONS’s Health Index for England

Dear Sarah 

Compliance review of ONS’s Health Index for England

We recently completed our compliance check of the Health Index for England against the  Code of Practice for Statistics. The Health Index is an important new development that provides an overall picture of health both for England as a whole and for regions and local authorities. The index is comprised of 56 indicators across a broad range of aspects of health within the three main domains of Healthy People, Healthy Lives and Healthy Places. This allows users to understand how health and its components has been declining or improving over time, and how it varies from area to area. The most recent release, published in November 2022, provides important insights into health in the earlier stages of the coronavirus (COVID-19) pandemic at national, regional and local authority level. 

Overall, our review found that the statistics demonstrate compliance with many areas of the Code of Practice. It is clear both from the published outputs and from talking to your team how much thought and hard work has gone into the development of the Index to date. We found a range of positive features, as well as some areas for improvement that we consider would enhance the trustworthiness, quality and value of the statistics. Addressing these areas will be beneficial as you work towards obtaining National Statistics designation for the Health Index. 

  • Both the main release and the local authority tool provide a good insight into how aspects of health have changed (or not) since the Health Index started in 2015. The explanations of the findings are clear, as well as possible reasons for some of the findings being provided. The improvements to the release for 2020 data reflect a lot of work from your team. 
  • It is good that you have collaborated with such a wide range of experts in the form of the Expert Advisory Group (EAG) in constructing the Health Index. We are also pleased that you have worked with The Alan Turing Institute to quality assure the Health Index methods, as well as having plans in place to develop the methodology further. This demonstrates a clear commitment both to ensuring that the methodology is sound, and to ongoing refinement of the methodology as you continue to develop the Index. We note that planned future developments include a health projections model and enabling local authorities to produce their own health indices. Your ongoing commitment to further developing the scope and utility of the Health Index is likely to benefit a wide range of users. 
  • It is evident from our conversations with your team how much consideration it has put into the various methodological aspects of the Index and how these might be taken forward, yet it is sometimes hard for users to see or appreciate this. Sharing details of both activities to date and planned work would help users to understand the amount of effort that is going into making the Index as useful and useable as possible. When sharing your plans, you should also provide details of how you plan to prioritise work going forward so it is clear to users when developments may occur. 
  • You should take into account uncertainty when presenting the statistics to help avoid over or under-interpreting. For example, you could explain changes more carefully, as there are some occasions where small differences are presented as though they are statistically significant, and some users might interpret these as being more meaningful than they really are. Your team told us that it has done a lot of work to try and understand how best to understand uncertainty in relation to the Index, and that it plans to continue to do so. It is fundamental to be clear to users about sources of uncertainty to help them in interpreting the findings appropriately. It would be helpful if you could bring together your work so far and publish, separately, some information on this and your plans for your users.  
  • Your team told us about its user engagement activities, and particularly the work that it has carried out at the local level to understand who uses the Index, what for, and what decisions it might inform. A notable example that the team told us about is specific NHS Trusts using the Health Index framework to develop a version at a lower geography to better understand and target health issues in their authorities. Expanding this excellent work to understand more about users and uses of the Index at national level and in the third sector will help to add further value. We welcome that your team is starting to develop its plans in this area.  
  • Although you have improved the timeliness of the most recent release compared to previous ones, it is still relatively out of date. It is good that you are looking to further improve the timeliness of the Health Index, potentially with an early provisional release, as well as exploring the trade-offs between timeliness and accuracy. Seeking users’ views on this possible provisional release as part of the planned user engagement work outlined above, would help you understand whether a more timely but possibly less accurate early release will be of benefit to them and make the Health Index more useful overall.  
  • Your team told us that it follows Reproducible Analytical Pipeline (RAP) principles in its work and gave strong examples of this. There are some areas where future developments could still be possible, such as automation of your quality assurance procedures or investigation of ways to semi-automate the factor analysis. We encourage you to keep this on your work programme and to continue to investigate where further improvements might be possible. 

 I would like to thank your team for their positive engagement on this review. Our Health and Social care domain will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.   

 Yours sincerely 

Mark Pont 

Assessment Programme Lead