Dear Alastair
Compliance Check of the Scottish Health Survey Statistics
We recently carried out a compliance check of the Scottish Health Survey (SHeS) statistics against the Code of Practice for Statistics. These statistics were designated as National Statistics in 2010, and we are pleased to confirm that they should continue to be accredited under the status now known as accredited official statistics (AOS).
The SHeS is a longstanding annual survey in Scotland providing data about the health of the population on a diverse range of topics including mental and physical health; diet and food security; and alcohol and smoking. It therefore provides a vital picture of the health of the Scottish population living in private households. It is used by the Scottish Government, Public Health Scotland and many health organisations, including charities for planning or for decision-making. The Scottish Parliament also regularly refers to the survey results when debating health issues. Endorsed by the Chief Medical Officer in Scotland, the SHeS has a Project Board consisting of stakeholders, the survey contractor and users which helps to increase confidence in the statistics. The Project Board also reviews the survey topic questions used each year to ensure that they remain relevant and are both appropriate and proportionate for the survey interviews, which span multiple age groups.
In carrying out our review, we consulted some users and took their views into account. We have set out our findings along with some recommendations in the annex that we consider would help to further improve the statistics. We will follow these up informally with the team as it works towards the next release.
I would like to thank your team for its positive engagement with us during this review. Please do not hesitate to get in touch if you would like to discuss any aspects of this letter.
Yours sincerely
Mark Pont
Assessment Programme Lead
Annex: OSR review findings and recommendations – Scottish Health Survey (SHeS)
Trustworthiness (T) findings
T1. The Chief Medical Officer for Scotland endorses the Scottish Health Survey (SHeS) in the Foreword. This acknowledgement credits the insights deduced by the collaborative efforts of the stakeholders involved in the production of the survey and the time given by the participants to inform future health policies. The team has a Project Board with representation from many key user organisations as well as other stakeholders closely aligned to the survey. The team sets out suggestions for changes and the Board reviews them and implements accordingly. This oversight group helps to increase user confidence in the statistics. Both of these examples demonstrate trustworthiness through transparent and independent decision-making and leadership.
T2. The Project Board decides on the questionnaires’ content, ensuring that the questions are relevant, the questionnaire lengths are appropriate for the varying age groups, and the respondent burden is minimised. The team undertook a questionnaire content review to learn about users’ views on the content of the SHeS. However, although the review responses summary has been published, the report summarising the updated questionnaire content is yet to be published so users remain unaware of any changes planned. We welcome the publication of this report in a timely manner, particularly as users reported that they were uncertain whether their suggestions had been taken forward.
Recommendation for improvement: T2
To make the decisions behind changes to the survey questions more transparent to users, the Scottish Government (SG) should ensure that the content report includes the criteria used to determine whether questions are continued, altered, added or removed each year. Where there are significant changes, these should be clearly highlighted and evidenced, establishing an audit trail to inform current and future users.
Quality (Q) findings
Q1. We found that there was much effort put into describing the processes used to recruit survey participants in detail and overcoming the challenges during Covid-19 in the Technical Report, all of which is helpful information for the reader. The fieldwork methodology report describes in detail how required sample sizes were achieved and how households were selected using the Postcode Address File.
However, the information on the processes for sample design and actual respondent engagement appears to be interspersed in some parts of the methodology. Thus, some users might find it challenging to distinguish and fully appreciate the technical aspects of the survey, such as:
- The recruitment procedures
- The sample estimation
- The collection and treatment of data
Recommendation for improvement: Q1
SG should seek feedback from a range of users about its methodology documentation and use this feedback to update the documentation accordingly to ensure it is as effective as possible.
Q2. We found that there are multiple references to the term ‘core questions’, which generally refers to all the questions asked in the relevant year to both adult samples. These references are further complicated by the fact that only a small number of questions included in the Scottish Health Survey, the Scottish Household Survey and the Scottish Crime and Justice Survey as part of the Scottish Surveys Core Questions are also referred to as the ‘core questions’. To aid user understanding, SG could clarify which set of ‘core questions’ are being referred to each time the term is mentioned to avoid any confusion between the different sets of questions.
Recommendation for improvement: Q2
As part of its wider user engagement, SG should determine whether users understand the references to ‘core questions’ throughout the documentation. If the similar terminology is found to confuse readers, SG could consider the use of alternative language that more clearly differentiates between the sets of questions.
Q3. The new dashboard provides 95% confidence intervals, which we heard was useful to many users. We found varying levels of the representation of uncertainty throughout the survey outputs, but the summary report doesn’t contain any information about uncertainty. Users told us that they would find it helpful to see confidence intervals in the supplementary tables as well.
Recommendation for improvement: Q3
SG should include confidence intervals in the supplementary tables to ensure users can better understand the quality of the estimates. SG could also review the presentation of uncertainty in other outputs and consider whether including further information about uncertainty might be helpful.
Q4. In the main report, we found that in some visual representations, the changes in attained values or scores can be ambiguous. For example, for the average mental wellbeing scores, the mean Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS) scores given on the associated Mental Health and Wellbeing chart are not clearly labelled or titled, and the decline in average scores between 2019 and 2022 could be open to misinterpretation. It is important that users clearly understand how to interpret the score changes from year to year in the time series analysis represented in charts.
Recommendation for improvement: Q4
To provide users with sufficient information to correctly interpret scores or values given in the chart, SG should ensure that all charts are clearly labelled with explanations of whether yearly fluctuations are statistically significant or not. This should increase the likelihood of accuracy of when data and charts are interpreted.
Q5. We noted that the team included a gender identity question between 2018 and 2021 onwards as part of the self-completion questions as this was an SG Scottish Survey of Core Questions (SSCQ) harmonised question. However, this question was removed from 2022 as it was no longer an SG SSCQ core question and was not being used for SHeS analysis purposes. The survey continues to include a question on sex (with possible responses ‘male’, ‘female’, ‘prefer not to say’) as this is an important characteristic for analysis of the survey results. The team told us that it would consult our recent guidance on Collecting and reporting data about sex and gender identity in official statistics when considering any future data collection changes in this area.
Value (V) findings
V1. In recent years, the SHeS has undergone some methodological changes, and during the Covid-19 pandemic in 2020, field work was suspended and a telephone survey was conducted instead based on a shortened version of the questionnaire and some Covid specific questions. The 2020 data were presented as experimental statistics due to differences in the profile and bias in the achieved sample compared to pre-Covid years and have not been included in the time series analysis since then. The absence of these data is clearly explained so that users are aware of why there is a gap in data tables, charts and on the dashboard.
V2. We were pleased to hear that the value of the SHeS data has been enriched by further data linking. The team told us that work done to link the Postcode Address File (PAF) data to the Community Health Index (CHI) database has increased the likelihood of identifying households with children under 16 making it a more efficient way to source child samples. There is also information explaining how the SHeS and health record data are linked and detailing the variables included. This will improve researchers’ access to the datasets, helping to obtain further valuable insights into the health of Scotland’s population.
V3. The objectives of the survey are clearly outlined in the main report. Users told us that they sometimes use other survey findings to complement the findings of the SHeS; for example, they might use the Alcohol Toolkit Study by the University College London when determining trends in alcohol use. Others mentioned that further information clarifying whether other sources and trends could be used or not would be helpful.
Recommendation for improvement: V3
To support SHeS users in considering how they might use SHeS statistics alongside other non-official sources, SG should provide further guidance, where practicable, about the extent to which SHeS data could be used alongside other reputable sources.
V4. SG has developed an innovative SHeS dashboard which provides comparative data going back to 2008 and is presented at national, local and health board levels. It is encouraging that RAP processes are being implemented and that further development work is planned. We heard that users have requested further breakdowns and were not always clear on whether they were being considered. It is encouraging to hear that SG intends to consult users on its plans to provide further breakdowns, such as employment type, urban/rural and health board/local authority data by the Scottish Index of Multiple Deprivation (SIMD).
Recommendation for improvement: V4
As SG develops its dashboard plans, it should communicate these so that users are more aware of the plans to progress their requests.
V5. Having comparable health statistics enables users to more easily observe trends across the UK countries. We found differences in the comparability and coherence of the survey across the survey topic areas that would hinder users in drawing such inferences.
Recommendation for improvement: V5
To provide further explanation on when statistics on a specific topic can be compared across UK countries, SG should include a more in-depth section on comparability. This will help to give users greater insight on the extent to which they can compare SHeS topic areas with other similar survey outputs in countries across the UK.
