M&A Note 3/2010

Introduction

1. In 2009 the UK Statistics Authority assessed four major population surveys which are carried out by the Scottish Government:
   • the Scottish Health Survey (1) ;
   • the Scottish House Condition Survey (2) ;
   • the Scottish Crime and Justice Survey (3) ; and
   • the Scottish Household Survey (4).The Authority designated all the outputs from these surveys as National Statistics conditional upon the Scottish Government implementing specific improvements and reporting them to the Authority within given timescales. It also found that that the surveys had different strengths and areas for improvement.
2. The results from these surveys – each of which has an annual cost of around £1.5 million – are used for a number of important purposes. Data from three of the surveys (Health, Crime and Justice, and Household) are used to measure progress against relevant indicators in the Scottish Government’s National Performance Framework5 . The results are also used to monitor and inform separate policy areas – for example:
   • the Scottish Health Survey is used by the Scottish Government and NHS Scotland to identify gaps in health service provision and to investigate the epidemiology of major chronic diseases and causes of death in Scotland;
   • the Scottish Crime and Justice Survey data can complement the separate police recorded crime statistics to provide a “rounded” picture of crime levels in Scotland. The survey results are used by the Scottish Government and police forces to make policy decisions about crime and justice which affect all the people of Scotland;
   • the Scottish House Condition Survey is used to monitor the uptake of energy efficiency measures, and to investigate the living conditions of vulnerable groups such as the elderly;
   • the transport section of the Scottish Household Survey is used to inform a range of policies, such as the concessionary bus fare scheme, and to assess the impact of transport on climate change in Scotland.
3. The purpose of this note is to summarise the main areas of good practice and areas for improvement which the Authority identified during the assessments, to help other producers of official statistics.Areas of good practice
   Coordination among the surveys
4. The survey managers have established a forum to share best practice. The four surveys are covered by the Scottish Government’s Long term strategy for population surveys in Scotland 2009-20196 which is overseen by the Scottish Population Survey Coordinating Committee. The strategy seeks to improve the way population surveys are run. A key element of this will be to introduce a standard set of socio-economic questions while retaining each survey as a distinct entity.Engagement with users
5. The Scottish Government carries out extensive user engagement for most elements of the Scottish Household Survey and adopts a range of approaches to maximise the use of survey data. These include:
   • presenting to a range of audiences on the uses of the survey data; and
   • providing user guides and training events for users to promote the use of survey data.
6. The Scottish Crime and Justice Survey was developed through extensive user engagement both within and outside the Scottish Government. The Scottish Government draws on expert advice from a technical working group which advises on ethical as well as technical issues for the survey. The group includes academics and members with experience of the British Crime Survey. This helps to promote comparability between the two surveys.
   
7.  The Scottish Health Survey website included pages listing uses and user views. The findings of user consultations and plans for future user engagement are available on the website.(7) The Authority regards this as good practice which should be adopted by the other surveys. Analysis and re-analysis of statistical data.
   
8. Unlike the other surveys, analysis of the Scottish House Condition Survey data was carried out by a team within the Scottish Government. The producers made the data more useful at local authority level by using recent administrative data on fuel prices, benefits and incomes to update estimates of fuel poverty derived from three years’ of data collection. They also published tables with data for all local authorities in response to data requests for individual local authorities.
   
9. Two of the population surveys have established positive links with postgraduate academic training. The Scottish Crime and Justice Survey is used as part of a methodology training course, and the Culture module of the Scottish Household Survey involves ESRC postgraduate students in data analysis and report drafting.
   
10. All four surveys allow users to re-analyse the data for their own purposes. The survey teams within the Scottish Government make datasets available to users in formats which allow them to re-use the data. They also provide information on various quality measures to allow users to decide if the survey data are suitable for their intended use. Where standard anonymised datasets do not contain all the information which users require, the Scottish Household Survey team provides a range of additional services:
    • an ad-hoc request service to respond directly to users’ requirements;
    • opportunities for researchers to use data samples for follow-up research – for example, on issues such as child poverty and bus use;8 and
    • provide users with special customised datasets when standard anonymised datasets do not contain all the information which users require.(9)
11. The Scottish Government has developed a simplified Scottish Household Survey dataset to stimulate the use of survey data, particularly among local authorities, voluntary organisations and academia. The structure of the data and the number of cases are the same as the full dataset but over 1,000 variables have been removed to facilitate data analysis. The data are formatted to provide all the necessary statistical functions for useful analysis and in a form that is easy to navigate10 . The Authority considers this a particularly helpful innovationAreas for improvement
    Pre-announcing forthcoming publication dates
12. The Scottish Government’s policy is to pre-announce the month of publication for statistical releases 12 months in advance and to pre-announce the actual date of release the month before the release. The list of forthcoming statistical publications is available on the Scottish Government website. However, the assessment team found that some survey managers were not clear about this policy and had delayed announcing a publication date until they were confident of the precise date on which the report would be published. The Authority’s Code of Practice for Official Statistics states that producers should publish the timetable for statistical releases twelve months in advance of publication.Reporting progress against targets
13. The Authority noted that there had been an inconsistent approach to the reporting of the Scottish Government targets in first releases across the different surveys. Although most national targets were included with appropriate signposting and commentary, some had been omitted from the survey publications. The Authority considers it helpful to users to report impartially and objectively about progress against targets which are based upon the statistics shown in statistical reports.Working with contractors
14. All four surveys involved contractor firms collecting and analysing the data, and in some cases, drafting the reports. In general, these arrangements worked well but in several cases there were problems with the quality of the data which the contractors sent to the Scottish Government. This led to delays in publishing the survey reports. The Authority suggests that the Scottish Government reviews the contractual arrangements with contractors to ensure that the initial quality assurance is carried out to the required standard.

---

1. See Assessment Report 10
2. See Assessment Report 11
3. See Assessment Report 12
4. See Assessment Report 26
5. http://www.scotland.gov.uk/About/scotPerforms
6. http://www.scotland.gov.uk/Topics/Statistics/About/Surveys
7. http://www.scotland.gov.uk/Topics/Statistics/Browse/Health/scottish-health-survey/Uses
8. http://www.scotland.gov.uk/Topics/Statistics/16002/FollowUpSurveys
9. http://www.scotland.gov.uk/Topics/Statistics/16002/SpecialDatasets
10. http://www.scotland.gov.uk/Topics/Statistics/16002/SHSLite