Findings

Introduction to the census in Scotland

The census is one of the most important sources of data and statistics, informing decisions about almost every aspect of life within the UK. It is of fundamental importance in allocating billions of pounds to local areas by the UK government and devolved administrations, as well as grants to voluntary sector organisations for example. The census helps every person in the UK get a better understanding of the places in which they live and work. 

The real value of the census will be realised on the release of census outputs. The National Records of Scotland (NRS) will have to deliver high quality data and statistics in a variety of forms to support the wide range of different uses required. It is essential that the data and statistics from the 2022 census in Scotland are reliable and provide valuable insights, meeting the rigorous standards of trustworthiness, quality and value outlined in the Code of Practice for Statistics.  

COVID-19 restrictions delayed or prevented NRS’s logistical planning and testing of census systems and processes. Necessary engagement with key stakeholder organisations was also affected. Consequently, following an options assessment process, and on the advice of NRS, Scottish Ministers made the decision to move the census date for Scotland to 2022. 

This meant Scotland’s Census was not aligned with equivalent exercises in England, Wales and Northern Ireland. The other censuses in the UK were undertaken in 2021, in the midst of the COVID-19 pandemic. Data from those reflect a unique snapshot in time, with topics such as employment and travel being unusual or changed from what might have been expected. The later-run Scotland’s Census was conducted at the time when most of the COVID-19 government restrictions had been lifted and life was returning to normal following the pandemic.  As such, the data collected on certain topics would still be changed from what might have been expected in the early stages of census planning, for example travel to work. 

This report has been written at a point in time where NRS is still in the process of data processing and quality assurance in the production of estimates of the population of Scotland.

Introduction to this assessment

The Office for Statistics Regulation (OSR) is carrying out phased assessments of the UK censuses produced by the Office for National Statistics (ONS), the National Records of Scotland (NRS) and the Northern Ireland Statistics and Research Agency (NISRA).  

This is the second assessment report for Scotland’s Census. Many of our considerations and judgements build on the preliminary findings as reported in our earlier assessment report published in October 2019. Our preliminary findings, and the last published update of progress – from September 2020 – can be found in Annex 1: Preliminary findings from the first phase of this assessment.  

Since then, we have continued to assess and advise NRS as part of the assessment process. We have collected and considered various sources of information to inform our judgements. We have taken account of the views of users and stakeholders who have shared their views with us on NRS’s work to date and its plans for outputs. We have continued to engage with NRS on its plans and progress as it moves from live census collection operations to data processing to deliver its census outputs. We have considered the openness and transparency of NRS through its public reporting and engagement with its stakeholders. We have also reviewed any materials published by NRS such as its report from January 2022 on how it considers it aligns with the standards of the Code.  

During this period, we have written publicly to NRS outwith our standard assessment reporting schedule where we considered a separate intervention was required. These include 

• August 2022, Mark Pont to Pete Whitehouse, Scottish Population Census: Transparency of plans and actions [with response from Pete Whitehouse] 
• November 2021, Ed Humpherson to Paul Lowe, OSR Publishes Assessment Reports for 2021 Censuses 
• September 2021, Ed Humpherson to Paul Lowe, Guidance for the sex question in Scotland’s Census [with response from Paul Lowe] 
• October 2020, Ed Humpherson to Paul Lowe, Guidance on the sex question in the forthcoming Census for Scotland [with response from Paul Lowe]. 

This assessment report focuses on the 2022 census in Scotland, produced by the NRS. OSR has identified areas of strength and good practice in NRS’s census planning and development and areas that NRS needs to address or where actions need to be taken. Our assessment will allow us to recommend whether the census outputs should be designated as National Statistics, in accordance with the requirements of the Statistics and Registration Service Act 2007, when they are first released.  

We have completed the equivalent phase of the assessment for 2021 Census in England and Wales and 2021 Census in Northern Ireland, confirming that these outputs can be designated as National Statistics. We expect to carry out a final review, which will involve an assessment of the census data and statistics in light of users’ views once all outputs have been published. 

Securing confidence in the data and processes

Plans for Scotland’s Census data collection and data processing

The census is the biggest statistical data collection exercise in Scotland and NRS has undertaken years of planning and preparation covering all aspects from statistical design to operational readiness. During this planning phase, in the run up to census day itself, NRS was transparent around its statistical methods development and its approach to data quality. It held various workshops for users and stakeholders on census topics, outputs and statistical methodology. NRS also  published materials including its external methods advisory panel papers and its statistical quality assurance strategy. 

This all led up to the significant milestone in NRS’s programme, the launch of its live data collection phase in February 2022. This phase saw millions of letters and correspondence sent to the people of Scotland to promote and enable the completion of either paper or electronic census questionnaires. Throughout this period NRS ran large-scale field operations and offered support to respondents in completing their census return, through running its contact centre for example. 

By the end of the planned collection period, while over two million household questionnaires had been received, the overall return rate and returns rates for local authorities were lower than had been hoped for. NRS published its Key Performance Indicators (KPIs) on levels of response and accuracy of its population estimates as at November 2019 (National estimates will achieve 95% confidence intervals +/-0.4%  bias <0.5%, local authority estimates will achieve 95% confidence interval +/- 3%). These set out a person response rate of at least 94% and person response rate in every local authority of at least 85%. At the end of April 2022, Scotland’s Census national return rate was at 79% and some local authority returns were below 70% including Glasgow City whose return rate was 66%. 

In response, on NRS’s recommendation, Scottish Ministers decided to extend the census collection period to the end of May 2022. During this time NRS posted regular updates on its website on how return rates were improving. NRS also made arrangements to move the timing of the operations for its Census Coverage Survey (CCS) and enhance the coverage of this survey. This was all done under extremely tight time scales and much public scrutiny. The assessment team recognises the extreme pressures faced by NRS staff at this time.  

In May 2022, NRS established an International Steering Group (ISG), chaired by Professor James Brown, the Australian Bureau of Statistics Professor of Official Statistics at the University of Technology, Sydney. The group’s membership includes census experts including the UK National Statistician – Professor Sir Ian Diamond and Professor David Martin, deputy director of the UK Data Service. The group was tasked with providing assurance that the census programme was ready to move on from its collect phase, and to provide direction to NRS on its statistical design following collect and supporting and advising NRS on its use of administrative data. We consider the establishment of this group to be extremely good practice to help secure the highest quality census data. 

By the end of the extension period, NRS, through its efforts, reported a national return rate of 87.9%, with 28 of Scotland’s 32 local authority areas achieving return rates of 85% or higher. The four local authorities where return rates were below the 85% level were: Dundee City (84.4%); Glasgow City (81.7%); Inverclyde (84.2%); and West Dunbartonshire (82.4%). On 2 June 2022, the ISG issued a statement of its confidence to proceed to the next stage of the census, referring to NRS’s position as having a ‘solid foundation’ on which to build. NRS moved into the next stage of its plans to undertake its CCS. This was completed in August 2022. 

OSR recognises that not all census forms had been receipted by the end of the collection extension period. NRS published a final set of achieved return rates as of 27 September 2022, once all forms had been receipted. This found further increases in return rates with 89.2% nationally and 30 of the 32 local authority areas achieving return rates of 85% of higher. 

Assuring users on methods and NRS’s confidence in data quality

While NRS worked to achieve the best census return rates possible, concerns were raised over the return rates and the impact on data quality. Most notably, the Scottish Parliament launched an inquiry into Scotland’s Census.  

Hearing from both NRS representatives and members of the International Steering Group, the Scottish Parliament’s Constitution, Europe, External Affairs and Culture Committee was presented with evidence and assurance on how data quality will be secured. These hearings reflect the position that the census return rates are not the only determinant of data quality but rather the combination of census, other data and effective estimation methodology. The Committee’s findings were published on the 17 November 2022. 

In a paper published in September 2022, NRS shared its high-level plans to secure high quality census estimates and committed to publish more-detailed information in Spring 2023 once adapted methods are finalised. NRS also plans to update its Statistical Quality Assurance Strategy which gives an overview of how NRS will measure the quality of the census programme.  

The ISG continues to meet regularly with NRS and has agreed to be involved in key decision points on methods development as work progresses. To further aid transparency, NRS has published the notes of actions from the ISG meetings so stakeholders and interested parties can see the types of discussion being held. 

The steps that NRS has taken to assure its users about the work it is doing to secure data quality are positive and we are confident that NRS will meet its commitments to publish further information for its users.  

NRS demonstrated its trustworthiness and was transparent in the early stages of census planning by providing users with its expectation of the quality for census outputs through, for example, its KPIs on accuracy. While specific quality measures will only be possible as part of its census outputs, NRS should be open ahead of those outputs about how its developments in data and methods are likely to affect the quality of estimates that will ultimately be produced.

Alongside the materials it has published on its work to secure data quality, we are aware that NRS has also spoken with a number of user groups and stakeholders about its plans. This has included engagement with a number of Scottish Government teams and with local authorities – through correspondence from NRS’s Chief Executive, updates to NRS’s established Population and Migration Statistics Committee, and individual meetings on request. NRS plans to continue its engagement activities over the coming months.  

When delivering future information and assurance for users on the expected quality of census data, NRS should consider different audiences with different levels of expertise. It should utilise a range of communication routes – such as website publications, newsletter updates, events or user meetings – as appropriate, to reach a broad range of users.  

Explaining the quality of census estimates and supporting use of data

When NRS publishes its census estimates from autumn 2023, the data and statistics will be accompanied by information to support users in their interpretation of the statistics. This may include quality indicators or guidance and caveats on topics where there are known quality issues. This section highlights a few examples of where this would be particularly important. 

Census topics – Sex and trans history

In a similar way to the census in England and Wales, Scotland’s Census included a long-standing question on sex and a new question to collect information on individual’s trans status or history (Note: Census in England and Wales did not ask a ‘trans status or history’ question rather one on individual’s gender identity) for the first time. In the development of these questions, NRS has published its research and user testing to support the decisions it has made for these questions, which has also been further discussed in the public domain through, for example, Scottish Parliament Committee hearings. 

There has been a high level of interest in this topic and both NRS and ONS, which runs the census for England and Wales, have been subject to legal challenge regarding the sex question and the guidance provided to respondents. In England and Wales, ONS made a change during census operations to its sex question guidance in response to an order from the High Court of England and Wales. In Scotland, following a judicial review, it was ruled that the guidance in Scotland’s Census did not permit, sanction, positively approve or authorise unlawful conduct. NRS made no subsequent change to the guidance it had put in place. 

As a result, while the Scottish census sex question and the response options have not changed and are consistent with those used in England and Wales, the guidance for respondents completing this question differs between countries. Scotland’s Census advised those who accessed the guidance ‘If you are transgender the answer you give can be different from what is on your birth certificate. You do not need a Gender Recognition Certificate (GRC).’ In England and Wales the guidance differed ‘If you are considering how to answer, use the sex recorded on your birth certificate or Gender Recognition Certificate’. Full guidance wording for the census sex questions can be found in Annex 2.

The guidance for the sex question in Scotland’s Census is in line with what was used for the previous 2011 census. This could therefore be viewed as unchanged between censuses. However, there is wider awareness of what this question is aiming to record, greater accessibility to the guidance material through use of online census questionnaires, and broader knowledge of the guidance material given the media attention on the topic for example. Additionally, the much closer proximity of the trans status question, in comparison to the ONS census equivalent, means that respondents may have a greater awareness of the possible differing approaches to defining sex within the census. All of which may mean respondent behaviours in answering the sex question may differ between 2011 and 2022.  

A number of users and stakeholder groups have raised concerns publicly, and directly with NRS, on the guidance for the census sex question. These include the wording of the guidance used in Scotland causing possible conflation between an individual’s sex and their gender identity which may affect the data collected from the sex question; a lack of comparability in census data across the UK and between censuses in Scotland; and any broader impact on statistics where other data collections may look to harmonise with the question used in Scotland’s Census. 

In order to maintain public confidence, NRS should respond to these concerns and the wider user need to understand the effect of the guidance on census responses. The Scottish Government should also reflect on how issues raised through census are relevant for its own guidance on Sex, gender identity, trans status – data collection and publication.  

Scotland’s new trans status or history question, a voluntary question asked of over 16s, has been welcomed by data users and stakeholders. It will be a very important new source of data on the size and makeup of the trans population in Scotland. Scotland asked a different question from that asked in England and Wales as a result of the outcome of separate question development processes. As with all census questions, NRS will explore and assess the data collected as part of its quality assurance procedures. It will be important that users are provided with accessible information and guidance to support users understanding of the strengths and limitations of the data and help them interpret the resultant statistics. 

NRS will be better placed to prepare for what information users may require based on the experiences from ONS as it publishes its census outputs on this topic. NRS is continuing to work with ONS on the development of its quality information for these and other census topics.   

Census topics – Veterans

The addition of the new veteran question will be significant in allowing the Scottish Government and the charity sector to monitor and support the veteran and armed forces population. In particular, household relationship data will allow users to explore the wider armed forces community (serving personnel, service leavers, veterans, and their families) as encompassed by the Armed Forces Covenant. A limitation of the census data collection, for NRS and indeed ONS, however is this will only be possible within households and veteran’s family members in different addresses will not be able to be identified in the same way. NRS should make clear the strengths and limitations of these data. 

As with other data, censuses offices will need to consider how to produce a harmonised UK estimate. In this case, for example, taking into account the differing order of response options between NRS and ONS. NRS will also need to support users in using this new data especially in combination with available Ministry of Defence data and the new veterans survey carried out by ONS. Many of the users may be academics or charitable groups who will have not engaged on previous censuses and so targeted proactive engagement may be necessary. 

The impacts of COVID-19

The 2022 census in Scotland was carried out at a time when life was still returning to normal following the COVID-19 pandemic and data users, such as local service planners, may need guidance on how best to interpret the data collected.  Unlike the 2021 censuses in the rest of the UK, topics such as employment, travel, and household status are less likely to be affected, however there may well still be unusual or changed data – in particular relating to travel to work for example.  

While NRS plans to explain any data quality issues related to COVID-19 alongside its census outputs, and despite some discussion in the International Steering Group, it has made little information available to users on this issue to date. We consider this would be helpful for users ahead of first outputs and would suggest NRS share its views on how census data may be affected by COVID-19 and to what likely extent.   

Supporting census data users 

NRS has yet to describe how the data quality of finalised census estimates will be assessed, measured and communicated to users, particularly given the changes it is making to its methods.  

It will be important for NRS to provide clear and accessible information to users about the quality of census data alongside all of its census outputs. This should cover all dimensions of quality including coherence and comparability.  In particular NRS should give attention to supporting users for new census topics for 2022, on key context changes for Scotland’s Census and in areas where users’ concerns may have been raised – for specific geographical areas, for certain population groups such as migrants, and on data about sex for example. 

Census estimates and UK comparability

NRS has worked closely with the ONS and the Northern Ireland Statistics and Research Agency (NISRA) as each office developed and implemented its census plans, and has committed to work collaboratively to understand and address the needs of UK census users. These commitments and the steps taken against these are publicly available through progress updates on the Conduct of the censuses across the UK, last updated in November 2020. This update outlines how the offices already work closely through a variety of networks and, led largely by ONS, have continued to engage with UK census users through a UK Data User Working group, which has included exploring the users’ needs through a survey of its members. 

Going into Scotland’s Census in 2022, there were known challenges with the comparability of census data across the UK. Most apparent of course was the difference in the timing of the censuses. There are also known differences in some census questions including the examples used in the previous section. These often reflect differing user needs identified through census development and research phases.    

While all the census offices have made positive steps when considering UK data users, little detailed information has been made public on how they will provide UK population estimates that take account of the 2022 census in Scotland. For example, NRS considers that its revised and adapted methods will not affect the comparability of census data across the UK, but this hasn’t been communicated to users. 

While responsibility for UK estimates lies with ONS, users are likely to reach out to both NRS and ONS for information and support and both will need to provide transparent, accessible and timely information on how UK population estimates will be provided.

Engaging with users

Engagement with users in the development of any census can be the largest, most wide–ranging engagement exercise undertaken by a statistical producer. NRS has facilitated large–scale formal consultation exercises, held external events and workshops and engaged on an individual basis with users and stakeholders. However, NRS has faced some challenges in securing satisfactory outcomes in some users’ opinions on contentious topic areas, as we described in our earlier assessment report.  

Communicating its plans for census outputs 

As NRS moves its focus and planning for Scotland’s Census to the release of data and statistics, it will be considering the needs of its users, captured primarily through its outputs consultation which closed on 10 February 2023. The consultation sought to capture users’ views on various areas related to the publication of census data including the sequence of topics published, the tools to publish data, and what users need from new census questions. The consultation was well advertised on social media and through communication routes including the Scotland’s Census stakeholder newsletter. NRS also ran a number of user events during the consultation period to outline its plans and answer users’ questions.  

NRS will assess how it responds to users’ needs and will publish details of its consultation outcomes. From the assessment team’s experience, we consider that whatever the specifics of the output plans, the most important aspect for users is NRS being clear on its release schedule in terms of content and timings, so users can schedule and plan the use of census data for their purposes accordingly. 

It will be important for NRS to confirm its arrangements and output release schedule, including the likely content and timing of first census outputs, at the earliest opportunity following the completion of its outputs consultation. Any changes to these plans should be clearly communicated to users. 

Sharing experiences and learning from Scotland's Census

Scotland’s Census has delivered various operational and technical successes including the development of new online data collection systems which supported an online-first approach to census – 90% of responses were submitted online in 2022. 

For Scotland’s Census, NRS has developed a Flexible Table Builder, a new dissemination tool for outputs which will allow users to create their own census data tables. As well as benefits for census data users, the development of the tool and the approach taken to applying statistical disclosure control within the new technology will be of interest to other statistical producers. NRS has also developed a new Scotland’s Census website. It has undertaken stakeholder engagement and various stages of user testing to improve the usability and accessibility of the website ahead of delivering its census outputs. NRS plans to improve its own website based on these developments. 

It is in no doubt, however, that NRS was faced with challenges in its collection phase. However, the steps that it has taken and how it has adapted its processes and methods in response should be commended. 

NRS’s experiences could have a significant part to play for future censuses, other survey data collections and the production of data and statistics using administrative data for Scotland, the UK and internationally. NRS should exploit all opportunities to share its learning with others. 

Maintaining strong leadership and management

As with all censuses, given the scale of operations and length of time involved in planning and preparation, retaining knowledge and expertise, given staff turnover for example, requires management. In recent months, both NRS’s Chief Executive and Director of Statistical Services have left the organisation. While interim arrangements have been put in place, for example NRS has arranged an ONS secondment into its census statistical leadership role, this is a significant change to the leadership and management within NRS. It will be important for NRS to ensure that it provides any necessary reassurances during this period of transition. 

In our previous assessment report, we encouraged the Scottish Government and NRS to review the seniority of the census statistical leadership role in NRS. We do consider this may help provide the required senior leadership and encourage better staff retention and progression for Scotland’s Census programme in the future. 

NRS should ensure that all reasonable steps are taken across the organisation to maintain business continuity while interim staffing arrangements are in place, so successful production of census data and statistics is not affected.