Actions taken and commitments made by NRS to meet the requirement:

• ‘Methodology Enhancements to Secure High Quality Census Outputs and Population Estimates’ paper, and accompanying short summary, published 8 June 2023. This paper describes a wide and varied range of research and analyses NRS has carried out in light of lower than expected return rates, to make decisions about updating statistical methods to ensure high quality census outputs. These methodological changes focus on enhancing methods related to coverage estimation, coverage adjustment and quality assurance and validation. NRS has also sought feedback from users on the detail provided in this paper to understand the demand for further information and plan for future publications. This feedback indicated that the level of detail currently provided meets user needs. NRS intends to publish further technical details once first census outputs are released.
• The foreword to this paper includes a statement from the International Steering Group (ISG), which was created to provide external advice for NRS on the quality of the census and planned statistical methods. The statement includes the line ‘These decisions were made on a sound methodological basis, providing NRS with a strong position for the statistical processing phase.’
• NRS has published a statement from its Interim Chief Executive, Janet Egdell, outlining her confidence in the methods. It has also published a letter from Janet to Alistair McAlpine, Chief Statistician at Scottish Government. In this letter, Janet sets out how NRS plans to demonstrate confidence in census statistics, which includes coherence with other statistics. Regarding coherence, Janet requests that Alistair reviews NRS’s assessment of coherence of the first outputs ahead of publication. NRS has published Alistair’s response where he agrees to convene a group of analysts and other experts to take this forward.
• NRS has worked in partnership with local authorities so they can directly contribute to the quality assurance of estimates for their own area ahead of first outputs. As explained in Janet Egdell’s letter, ‘ahead of publication, we will send aggregate estimates for their area so they can validate against other locally held sources. We will investigate all inconsistencies to understand whether there is evidence that revisions are required’. At the start of the exercise, NRS held two webinars with local authority contacts to clearly outline their expectations for participants’ contributions; to make use of additional local evidence and feedback any inconsistencies for NRS to investigate further. 32 local authorities in Scotland were invited to take part in the exercise, with 28 actually participating. All local authorities were also offered a meeting with NRS to discuss the estimates. NRS also requested meetings with local authorities in specific circumstances, for example those with the lowest census return rate and those with the biggest inconsistency with mid-year population estimates. NRS considered all feedback and prioritised follow up action in terms of the severity of the issue and the strength of the evidence provided. Some of this further investigation resulted in adjustments and some did not. NRS intends to provide further information about this exercise in the quality report that will be published alongside the first release.it
• NRS carried out an outputs consultation to gather user feedback on its plans for Scotland’s Census 2022 outputs. It received 146 responses to the consultation from a wide range of organisations including central and local government, academic institutions and charity organisations. The outputs consultation report was published in May 2023 setting out responses to the feedback and changes that were made to plans for outputs, as well as any further investigation they intend to carry out, following the consultation.
• NRS has published an outputs schedule to inform users of the timing of publications and what it intends to include in each release.
• NRS has published a Statistical Quality Assurance Strategy where it outlines ‘critical success factors’ for achieving high quality results. NRS plans to provide more up-to-date information about these within the quality report that will be published alongside the first release of outputs.
• NRS has communicated all of this information via its monthly newsletter, and discussed these topics at webinars in June 2023. At the webinar we attended, NRS was very open and shared some reassuring messages around wanting to be transparent, answer users’ questions, and for users to feel confident in the estimates that are produced.
• OSR has had sight of the materials used to communicate updates to the methodology and quality assurance at meetings with key user groups. These materials also address the consultation and any changes that were made as a result of user feedback.

OSR’s evaluation of evidence:

As noted in the assessment report, while NRS worked to achieve the best census return rates possible, concerns were raised over return rates and the impact on data quality. Subsequently, NRS established an International Steering Group and published a paper that shared its plans to secure high quality census estimates. The group is made up of pre-eminent international authorities in census, coverage and administrative data, including individuals with direct experience of similar response issues. In taking forward the changes in its approach, NRS also carefully considered which administrative records to use to supplement census data, and worked closely with administrative data owners to ensure it had access to all sources identified by the ISG as ‘essential’. The assessment report noted these positive steps, but also that NRS should be open ahead of first outputs about how its developments in data and methods are likely to affect the quality of estimates that will ultimately be produced.

Following the assessment report, NRS has taken a transparent approach to providing information and assurance for users regarding the quality of census data ahead of first outputs. It has proactively considered how to provide this assurance and communicated this information in a number of different ways for different audiences.

NRS has fed back to us that the local authority quality assurance work was a useful exercise and particularly highlighted the value of engaging with local authorities to provide further explanation of how the census estimates relate to other sources. NRS shared with us its approach to the exercise, including how it would use the findings to further improve Census estimates, and we are content that NRS has been open to the inconsistencies identified and investigated these appropriately.

NRS has proactively identified users’ needs via the outputs consultation and is open about subsequent changes to its approach in the report. It is important that NRS continues to listen to any user feedback on the assurances it provides or its plans for future outputs, in order to evaluate the need for further information and respond to these needs. We welcome NRS’s plans to publish further technical information following the publication of the first outputs.

For this requirement, we are content that NRS has taken all appropriate actions that we would expect in order to address this requirement.

Actions taken and commitments made by NRS to meet the requirement:

• NRS’s engagement plans are in three phases: ahead of the first release of outputs, at first release, and after first release.
• Prior to first outputs, NRS has targeted engagement at key population data users, such as the Scottish Fiscal Commission and HM Treasury, to discuss why they can be confident in, and trust, Scotland’s Census 2022 statistics. It has also engaged with specific groups such as the Population and Migration Statistics Committee, the minutes of which are published.
• NRS has sought feedback from initial meetings and subsequently reviewed its approach, identifying other stakeholders to engage with and refining its messages.
• NRS has worked directly with local authorities on the quality assurance of census data, and is also engaging with local authorities via broader groups and networks such as the Convention of Scottish Local Authorities.
• NRS’s wider stakeholder engagement included webinars held at the end of June 2023, which were open to all and publicised via newsletters, mailing lists, emails to specific user groups and social media.

OSR’s evaluation of evidence:

As noted for the previous requirement, following concerns over return rates and the subsequent impact of this, NRS took a number of steps to assure users about its work to secure data quality. In addition to delivering information and assurance for users on the expected quality of census data ahead of first outputs, the assessment report also specified that NRS should consider different audiences with different levels of expertise. It also stated that NRS should utilise a range of communication routes – such as website publications, newsletter updates, events or user meetings – as appropriate, to reach a broad range of users.

NRS has taken a thorough and measured approach to user engagement ahead of first outputs, and has carefully considered its communication in order to help users understand key messages.

It will be important for NRS to continue to review and refine user engagement plans following the release of first outputs, in response to feedback and in preparation for future releases; for example, we would expect future engagement to include user groups relevant to the specific census topic outputs in summer 2024.

We are content that NRS has taken all appropriate actions that we would expect in order to address this requirement.

Actions taken and commitments made by NRS to meet the requirement:

• NRS’s first census outputs will include the population by age and sex at Scotland and local authority level. Regarding sex, NRS has committed to provide information about definitions, question guidance and the quality of this data alongside these first outputs via a Quality and Methodology Information (QMI) report. In this report NRS also plans to address the coherence and comparability of these statistics with other censuses.
• Outputs on specific census topics, including trans history, are expected to be published in summer 2024. NRS intends to publish further information about the trans history question, including definitions, guidance, and the quality of the data at that time.
• NRS is also aware of recent issues concerning the gender identity question included by ONS in the 2021 Census for England and Wales. Although NRS’s question is different, NRS is liaising with ONS on its findings and its publication plans.

OSR’s evaluation of evidence:

In a similar way to the census in England and Wales, Scotland’s Census included a long-standing question on sex and a new question to collect information on individual’s trans status or history (Note: Census in England and Wales did not ask a ‘trans status or history’ question, rather one on individual’s gender identity) for the first time. There has been a high level of interest in these topics and both NRS and ONS, which runs the census for England and Wales, have been subject to legal challenge regarding the sex question and the guidance provided to respondents. As a result, while the Scottish census sex question and the response options have not changed and are consistent with those used in England and Wales, the guidance for respondents completing this question differs between countries (ONS made a change during census operations to its sex question guidance).

Therefore, the assessment report highlighted a few examples of where it would be particularly important to provide information alongside the data and statistics to support users in their interpretation of the statistics, and this included sex and trans history.

NRS has committed to provide appropriate quality information alongside first outputs and more detailed information on specific census topics in summer 2024. We are confident that NRS will meet its commitments to deliver this information across the coming year.

We welcome NRS’s engagement with ONS on the trans history/gender identity questions and expect this engagement to continue ahead of the release of NRS’ trans history outputs in summer 2024.

Actions taken and commitments made by NRS to meet the requirement:

• NRS shared with us its plans for communicating uncertainty within the first release. NRS intends to provide uncertainty information upfront, and include useful, practical information for users about how to understand, interpret and use census estimates. NRS plan to provide more-detailed information towards the end of the publication that will include confidence intervals, as well an explanation of what these mean in order to help users interpret them.
• NRS intends to publish a QMI report alongside first outputs in September 2023, including detailed information about quality assurance of processes, validation of the estimates, and uncertainty (such as confidence intervals). NRS plans for this QMI report to also include information about the context of the pandemic and how census data may have been affected.
• Ahead of first outputs, during user engagement NRS was clear in its messaging that the final statistics are estimates and will therefore have some degree of uncertainty.
• NRS plans to provide further quality information later in the outputs schedule; for example, when outputs on individual census topics are published in summer 2024, which will include appropriate quality information for new topics such as veterans. It also plans to link quality information to the Flexible Table Builder that it intends to make available for these later outputs.
• For sex, NRS intends to provide quality information such as metrics for access to guidance, the level of item missingness for the question and comparability with 2011 and other UK censuses.

OSR’s evaluation of evidence:

The assessment report noted that NRS had yet to describe how the data quality of finalised census estimates would be assessed, measured and communicated to users, given the changes it has made to its methods. It also stated that it would be important for NRS to provide clear and accessible information to users about the quality of census data alongside all of its census outputs, covering all dimensions of quality including coherence and comparability. The report stated that in particular NRS should give attention to supporting users for new census topics for 2022 (such as veterans), on key context changes for Scotland’s Census (such as any data quality issues related to the pandemic) and for known areas of user interest (for example for data about sex).

NRS has committed to publishing information about uncertainty within the first release, as well as a QMI report alongside the main publication. Following conversations with NRS, we are confident that these will include relevant information to help users understand data quality.

We are also content with NRS’s plans to provide further quality information in due course, and in different ways, such as via the Flexible Table Builder. We expect NRS’s quality documentation to be expanded on as additional information and needs emerge.

We are content that NRS has taken all appropriate actions that we would expect in order to address this requirement.

Actions taken and commitments made by NRS to meet the requirement:

• While responsibility for UK estimates lies with the ONS, NRS continues to work with the ONS and NISRA regarding the approach to communicating plans for UK census data.
• NRS has committed to state its commitment to working with the ONS and NISRA on UK census data in the QMI published with first outputs. Here, NRS also intends to state its position that its revised and adapted methods will not affect the comparability of census data across the UK.

OSR’s evaluation of evidence:

The assessment report notes that NRS has worked closely with the ONS and NISRA as each office developed and implemented its census plans, and has committed to work collaboratively to understand and address the needs of UK census users. These commitments and the steps taken against these are publicly available through progress updates on the Conduct of the censuses across the UK, last updated in November 2020. This update outlines how the offices already work closely through a variety of networks and, led largely by ONS, have continued to engage with UK census users through a UK Data User Working group, which has included exploring the users’ needs through a survey of its members.

The assessment report acknowledges that going into Scotland’s Census in 2022, there were known challenges with the comparability of census data across the UK; particularly the difference in the timing of the censuses. There are also known differences in some census questions, often reflecting differing user needs identified through census development and research phases. The report highlights that while responsibility for UK estimates lies with the ONS, users are likely to reach out to both NRS and the ONS for information and support, and therefore both will need to provide transparent, accessible and timely information on how UK population estimates will be provided.

NRS intends to continue working with the ONS and NISRA on UK census data and plans to state this commitment publicly alongside their first outputs. We look forward to the results of the cross-nation collaboration.

We are content that NRS has taken all appropriate actions that we would expect in order to address this requirement.