Dear Sarah
Compliance review of ONS’s Health Index for England
We recently completed our compliance check of the Health Index for England against the Code of Practice for Statistics. The Health Index is an important new development that provides an overall picture of health both for England as a whole and for regions and local authorities. The index is comprised of 56 indicators across a broad range of aspects of health within the three main domains of Healthy People, Healthy Lives and Healthy Places. This allows users to understand how health and its components has been declining or improving over time, and how it varies from area to area. The most recent release, published in November 2022, provides important insights into health in the earlier stages of the coronavirus (COVID-19) pandemic at national, regional and local authority level.
Overall, our review found that the statistics demonstrate compliance with many areas of the Code of Practice. It is clear both from the published outputs and from talking to your team how much thought and hard work has gone into the development of the Index to date. We found a range of positive features, as well as some areas for improvement that we consider would enhance the trustworthiness, quality and value of the statistics. Addressing these areas will be beneficial as you work towards obtaining National Statistics designation for the Health Index.
- Both the main release and the local authority tool provide a good insight into how aspects of health have changed (or not) since the Health Index started in 2015. The explanations of the findings are clear, as well as possible reasons for some of the findings being provided. The improvements to the release for 2020 data reflect a lot of work from your team.
- It is good that you have collaborated with such a wide range of experts in the form of the Expert Advisory Group (EAG) in constructing the Health Index. We are also pleased that you have worked with The Alan Turing Institute to quality assure the Health Index methods, as well as having plans in place to develop the methodology further. This demonstrates a clear commitment both to ensuring that the methodology is sound, and to ongoing refinement of the methodology as you continue to develop the Index. We note that planned future developments include a health projections model and enabling local authorities to produce their own health indices. Your ongoing commitment to further developing the scope and utility of the Health Index is likely to benefit a wide range of users.
- It is evident from our conversations with your team how much consideration it has put into the various methodological aspects of the Index and how these might be taken forward, yet it is sometimes hard for users to see or appreciate this. Sharing details of both activities to date and planned work would help users to understand the amount of effort that is going into making the Index as useful and useable as possible. When sharing your plans, you should also provide details of how you plan to prioritise work going forward so it is clear to users when developments may occur.
- You should take into account uncertainty when presenting the statistics to help avoid over or under-interpreting. For example, you could explain changes more carefully, as there are some occasions where small differences are presented as though they are statistically significant, and some users might interpret these as being more meaningful than they really are. Your team told us that it has done a lot of work to try and understand how best to understand uncertainty in relation to the Index, and that it plans to continue to do so. It is fundamental to be clear to users about sources of uncertainty to help them in interpreting the findings appropriately. It would be helpful if you could bring together your work so far and publish, separately, some information on this and your plans for your users.
- Your team told us about its user engagement activities, and particularly the work that it has carried out at the local level to understand who uses the Index, what for, and what decisions it might inform. A notable example that the team told us about is specific NHS Trusts using the Health Index framework to develop a version at a lower geography to better understand and target health issues in their authorities. Expanding this excellent work to understand more about users and uses of the Index at national level and in the third sector will help to add further value. We welcome that your team is starting to develop its plans in this area.
- Although you have improved the timeliness of the most recent release compared to previous ones, it is still relatively out of date. It is good that you are looking to further improve the timeliness of the Health Index, potentially with an early provisional release, as well as exploring the trade-offs between timeliness and accuracy. Seeking users’ views on this possible provisional release as part of the planned user engagement work outlined above, would help you understand whether a more timely but possibly less accurate early release will be of benefit to them and make the Health Index more useful overall.
- Your team told us that it follows Reproducible Analytical Pipeline (RAP) principles in its work and gave strong examples of this. There are some areas where future developments could still be possible, such as automation of your quality assurance procedures or investigation of ways to semi-automate the factor analysis. We encourage you to keep this on your work programme and to continue to investigate where further improvements might be possible.
I would like to thank your team for their positive engagement on this review. Our Health and Social care domain will continue to engage with you and your team on progress in the coming months. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter further.
Yours sincerely
Mark Pont
Assessment Programme Lead