Findings

Introduction

The Census is one of the most important sources of data and statistics, informing decisions about almost every aspect of life within the UK. It is of fundamental importance in allocating billions of pounds to local areas by the UK government and devolved administrations, as well as grants to voluntary sector organisations for example. The Census helps every person in the UK get a better understanding of the places in which they live and work.

The real value of the Census will be realised on the release of Census outputs. The Office for National Statistics (ONS) will have to deliver high quality data and statistics in a variety of forms to support the wide range of different uses required. It is essential that the data and statistics from the 2021 Census in England and Wales are reliable and provide valuable insights, meeting the rigorous standards of trustworthiness, quality and value outlined in the Code of Practice for Statistics. The 2021 Census was taken during a national COVID-19 lockdown in England and Wales and, as such, captured data on individuals and households at a unique period of time. Consideration and understanding of the quality and value of Census data and how users’ needs may or may not be met will be especially important in these circumstances.

This is the second assessment report and many of our considerations and judgements build on the preliminary findings as reported in our earlier assessment report published in October 2019. Our preliminary findings, and the last published update of progress – from September 2020, can be found in Annex 1: Preliminary findings from the first phase of this assessment. Since then, we have worked closely with ONS to understand how it has taken forward actions based on our earlier findings and progressed its plans to deliver high quality, valuable Census outputs. This report identifies areas of strength, good practice and innovation in ONS’s Census planning and development as well as identifying some areas where improvements need to be made.

As with the earlier phase of this assessment, ONS published a report providing an update on its progress and how its plans and developments align with the standards of the Code. The approach of sharing and explaining this publicly echoes the key aspects of the Code, such as transparency and accountability, and the assessment team recognises this positive communication with stakeholders.

Understanding the context for Census 2021

Communicating the impacts of COVID-19

The 2021 Census in England and Wales was undertaken at an unprecedented time. ONS had to adapt and prepare for live Census operations during the COVID-19 pandemic. It had to take steps to ensure the safety of the public and of its field force staff, as described in a public statement issued in January 2021. ONS undertook a variety of scenario planning and contingency activity from early on in the pandemic covering operational and data risks. It also provided additional guidance to respondents on how to complete certain Census questions, for example for students where additional support was available through a special student webpage and targeted communications on Census were taken forward.

Undertaking Census during COVID-19 meant that data on some topics such as employment, travel, and household status may well be unusual or changed. Census data will reflect the population of England and Wales at this rather unique point in time and data users’ needs, such as those of local service planners, may no longer be met from the data collected in 2021.

ONS is currently undertaking Census data processing and quality assurance activities to understand and better assess the quality, strengths and limitations of the Census data collected in March 2021. ONS also told us of the work of a taskforce set up to investigate the impact of COVID-19 on data sources including Census and administrative data sources used more widely in ONS. The assessment team recognises the important work being undertaken to gather these insights.

We recognise that through its outputs consultation, which closed on 6 October 2021, certain information has been shared on the impacts on Census data and ONS has asked questions of users about their data needs. ONS has also held some topic specific user events, for example in September 2021 ONS ran a webinar on Census 2021 Plans for international migration and origin-destination outputs. At this event, ONS shared some of its thinking about the likely impact on data, outputs, and meeting any gaps in provision on these topic areas. Sharing and getting user input throughout ONS’s development of Census outputs – even where specific solutions or commitments to deliver cannot yet be made – keeps users and stakeholders informed and lets them know that their concerns are being taken forward. This event was a good example of ONS keeping users who attended the session informed of its thinking.

Given the interest from a range of data users to understand how their needs may or may not be met, it is regrettable this information – on topics affected by COVID-19 or on other factors such as the UK leaving the European Union – is not replicated or readily available on ONS’s Census webpages.

ONS has made little information widely available on the steps it is taking to investigate, or otherwise meet, user needs for data affected by COVID-19. This is a significant gap in the assurances offered by ONS on its plans to address impacts on data. Furthermore, ONS has told us it is unlikely to publish more information until the completion of the outputs consultation and once it has a firmer understanding of data impacts.

Census and population estimates for UK

In July 2020, National Records of Scotland (NRS) announced that Scotland’s Census would be moved by a year to March 2022 due to the impact of COVID-19. This difference of Census dates across the UK will have further impacts on population data, in particular for UK Census data and plans for how UK population estimates for 2021 will be provided.

The three Census offices – ONS, NRS and NISRA – have worked closely as each office has developed its Census plans and have made commitments to work collaboratively to understand and address the needs of UK Census users. These commitments and the steps taken against these are publicly available through progress updates on the Conduct of the censuses across the UK, last updated in November 2020. This update outlines how they already work closely through a variety of networks and, led largely by ONS, have continued to engage with UK Census users through a UK Data User Working group, which has included exploring users’ needs through a survey of members.

While ONS and the other Census offices have made positive steps when considering UK data users, little detailed information has been made public on the availability, comparability and limitations of affected Census data for England and Wales or at a UK level – or how they will provide UK population estimates for 2021. ONS has shared with us its thinking regarding its communication plans around 2021 UK population estimates and we acknowledge that ONS has made assurances that 2021 estimates will be available, for example in its Population and migration statistics system transformation – overview published in June 2021. We welcome the ONS team’s intention to provide iterative updates to users as it becomes more certain of its position.

Further steps need to be taken by ONS to communicate plans and provide more-detailed information, when available, to users of UK population estimates and UK Census data in particular.

Enhancing confidence in the data and processes

Handling personal data

Data collection by government bodies and the public acceptability or willingness of individuals to share data for significantly important collections such as Census has been of growing concern over recent years. ONS made available a variety of resources to support individuals’ understanding and offer reassurance about its appropriate handling of data. ONS published clear and prominent information on the Census 2021 website for Census respondents regarding what their data will and won’t be used for on ‘Your data and security’ webpage during live Census operations.

ONS also commissioned information assurance reviews undertaken by a cyber security consultancy. The most recent review found ‘strong controls were also in place to detect and respond to threats that may impact the Census when it is in live operation. This 2020 assessment found that security controls in place have built upon and enhanced those in place during the 2019 rehearsal’.

ONS has established frameworks and protocols covering the various aspects of data governance and secure data handling when preparing data and statistics, for example ONS’s organisational data protection policy and data strategy can be found on its website. For Census statistical outputs available publicly, ONS has developed and tested disclosure control methods to be applied to protect the confidentiality of individual respondents. ONS will also make secure Census microdata available for more-expert data users, for example through its Secure Research Service which gives access only to approved researchers in a safe setting to protect data confidentiality.

Census data collection

The quality of Census estimates is dependent on the data submitted by individuals and households during the main collection phase for Census, both in terms of response rates and accuracy. From March to July 2021, ONS ran Census collection operations in England and Wales. As reported in its press release on 4 May 2021, 97 percent of occupied households had completed Census returns, with all local authority areas seeing over 90 percent occupied household return rates. This exceeded the success criteria as set out it the White Paper Help Shape Our Future: The 2021 Census of Population and Housing in England and Wales – 94 percent nationally and 80 percent locally in all local authority areas. This is a positive signal of a successful collection phase.

ONS has completed its Census Coverage Survey (CCS) – an independent survey carried out after main Census collection to identify over and under coverage – and its Census Quality Survey (CQS) – a separate survey which asks the same questions asked in Census to assess how accurately Census questionnaires have been completed.

ONS told us the response to the CCS was lower than expected but, given the higher-than-expected return rates for the Census itself, it is confident that the CCS will enable it to produce accurate population estimates. In order to fully mitigate against any risk to greater variance or bias in the final population estimates arising from this, ONS is developing a new approach to be able to measure any bias using administrative data. This approach has been subject to external scrutiny by its Methodology Assurance Review Panel in October 2021.

On 4 October 2021, ONS published two articles on Census operations, Designing a digital-first census and Delivering the Census 2021 digital service. The 2021 Census was the first digital-first Census in England and Wales. This approach hoped to achieve improved data collection in terms of both data quality – through question routing, data validation and access to online guidance – and efficiency of operations – including faster data processing and reduction in costs. These articles contained information on the digital-first approach and data on how the approach was delivered in live operations as well as technical information aimed at digital professionals across government. We would expect to see further data and information made available for users and the wider statistical community on, for example, if the application of the new approach would affect quality when comparing data over time or for geographies where a greater proportion of paper questionnaires were used.

Assuring data quality

As ONS has completed its collection phase, its focus has moved into processing and estimation of Census data. ONS’s Statistical design for Census 2021, published in October 2020, helpfully set out for users and stakeholders the various stages of Census delivery and how it ensures the data is of sufficient quality. This is a useful document which helps explain the scale and complexity involved in producing Census estimates and helps reinforce to users that the data at the end of the day are estimates and not purely counted Census submissions.

In January 2020, ONS published its Approach and processes for assuring the quality of the 2021 Census data. This report detailed the quality assurance strategy employed and explains how ONS will validate Census estimates through plausibility checks, comparisons with other data sources and use of expert topic-based knowledge. The assessment team discussed with ONS its use of administrative data-based population estimates and other administrative data sources within the quality assurance process alongside the use of administrative data within Census processing. ONS is confident that it is managing the risk of circularity that can arise when using administrative data sources both to account for missing data and for quality assurance purposes.

ONS has continued to develop and evolve its quality assurance strategy. For example, it has enhanced its quality assurance through greater involvement of local authorities in the process. ONS has always worked closely with local authorities to discuss proposed approaches to quality assurance and utilise their expertise. For Census 2021, ONS will be providing local authority representatives with limited access to anonymised, provisional Census estimates for their area, strictly for the purposes of quality assurance prior to publication. This is a positive addition to the assurance process and reflects ONS’s commitment to quality.

ONS also outlines, in its Approach and processes for assuring the quality of the 2021 Census data, what information on quality it expects to publish alongside Census data, including confidence intervals for national and local authority population estimates and descriptions of the size and nature of errors introduced through Census data processing. It is helpful that ONS has signalled its intention to make this information available to users.

For some aspects of Census developments, data quality concerns have been raised by some users, for example on data collected for the Census sex question. This was heightened given that ONS made a change to its guidance during live Census collection operations. The change was made in response to a court order, issued by the High Court of England and Wales, following a challenge to the wording of the guidance.

ONS told us that it expects that the impact of changing the guidance will be minimal. To evidence this, provide reassurance and give confidence to users in the quality of its outputs, ONS needs to mitigate for this in the production of Census results and explain to users the impact of the change on Census data including at disaggregated levels. For example, ONS might publish evidence on how many respondents accessed the guidance – before and after the change – and if, through data processing, it becomes aware of any issues with the data on sex collected.

Developing Census methods and use of administrative data

ONS takes a structured and diligent approach to ensure Census methods and processes are robust and rigorous. Methods development involves review at its internal Census Research Advisory Group before being taken to the Methodology Assurance Review Panel (MARP). This panel consists of external independent experts who also provide feedback to the National Statistician. The assessment team attended a MARP meeting where we witnessed the degree of challenge offered by this group and the openness of ONS to constructive review. The MARP papers and minutes of the meetings were available in the public domain during the earlier stages of Census developments, which offered a greater level of transparency and assurance of the methods development process. However, papers and minutes have not been published since January 2021; ONS should engage with the relevant panel secretariat to ensure these papers and minutes are published as soon as practicable.

ONS’s work to develop appropriate statistical disclosure control (SDC) methods is one example of this approach. ONS has been transparent about its work to date at user events and has provided more detail through published MARP papers and minutes. ONS has continued to develop its methods based on feedback and through various data rehearsals undertaken earlier in the year. ONS’s final plans on disclosure control methods and parameters were discussed at a MARP meeting in August 2021, before being approved by the UK Census Committee in October.

SDC is essential for protecting personal data but can be challenging to explain to users. ONS is preparing both technical and more user-friendly materials to support user understanding and we have suggested that ONS work with experts on how best to present, explain and make accessible information on quality, uncertainty and SDC methods to a range of expert and non-expert users, taking into account the range of statistical outputs and dissemination tools that will be available for users.

ONS uses administrative data sources for a range of purposes in Census from supporting data collections, quality assurance and for output geographies. Most notable for 2021 Census is the use of Valuation Office Agency (VOA) property attribute data to produce new outputs replacing the number of rooms question asked in previous Censuses. ONS has published various research on the use of VOA data, including Admin-based statistics for property floor space and Admin-based statistics for property type feasibility research, and in July 2020 ONS also published a quality assurance of administrative data used in Census document for VOA property attribute data. Having this information and research available to stakeholders while ONS was still finalising its plans for use of this data is good practice, aids transparency and facilitates greater input from a wider expert audience.

ONS has assessed the strengths and limitations of introducing these administrative data to replace an existing Census question and judges this will be of sufficient quality to support topic analysis and other outputs on housing and over and under occupancy measures. This is just one example of where providing users with information and assurances on the strengths and limitations of the source data, including any impacts on definitional differences and any discontinuity in the data, will be extremely important.

Data, quality and methods

Information on quality, strengths and limitations and being clear what users can and cannot do with the data helps support appropriate use of the statistics. This will be particularly important for 2021 Census data when it comes to areas of change, for example on data particularly impacted by COVID-19 or for new Census questions. ONS has a responsibility to support and assure users of the quality of Census data and its plans to deliver this alongside Census outputs.

To assure data users of the value and quality of Census data, ONS should ensure its plans to provide information on quality – including information on data collection and processing, quality assurance activities and quality measures, methods and use of administrative data, and ONS’s judgement on appropriate use of Census data – are delivered.

Responding to user needs

User engagement opportunities

ONS has continued to make use of public events and email updates as a way of keeping users informed of Census plans and developments. This composite approach has helped extend the reach of user engagement beyond expert users and stakeholders, delivering a broader message of how Census data serve the public good. For example, ONS has recently undertaken its Census output consultation which ran from July until October 2021. This has been an important exercise to better understand users’ needs and inform outputs planning. The consultation was accompanied by a series of events to support understanding of the areas of focus and the process for prioritising outputs, as well as to encourage responses to the consultation itself. ONS reported that it received over 300 responses to its consultation from a range of users including local authorities, charities, community groups and commerce.

Having open and effective dialogue with users and stakeholders is vital in demonstrating that ONS is a trustworthy organisation that actively listens and responds to users’ views. As with all engagement exercises run for Census, or more widely across ONS, experiences – both good and bad – can be learnt from to improve and build on engagement activities. For example, as part of this assessment, we spoke with some stakeholders who raised concerns or issues they faced when engaging with ONS in relation to the Census sex question. As mentioned in our Phase 1 assessment report, the development of the question and related guidance on the Census sex question has proved challenging for ONS and at times has been contentious, with user and stakeholders holding strong and conflicting views.

While we recognise that ONS took steps to engage with a range of stakeholders and commissioned further work in response to challenges on its approach and position, we consider confidence in ONS’s trustworthiness has been undermined with some of these users because elements of its communication with individuals, and publicly, could have been handled better.

In summer 2020, ONS held a stakeholder roundtable event to hear the different and often opposing viewpoints from stakeholders on the guidance for the Census sex question. Following that event ONS took forward work to further research and explore issues raised. While we are aware ONS provided some updates to the stakeholders as its research was taken forward, we also consider that these updates could have been fuller and more timely to better keep stakeholders updated on its progress and plans for communicating outcomes. We also consider that there was some confusion as to the level of stakeholder involvement in this research and decision making, which may have contributed to differing expectations of involvement.

Additionally, prior to the time when ONS made public its decision on the wording of the sex question guidance in February 2020, the assessment team was told by some users that in its more formal or public external communications, ONS’s messaging seemed confused and was unclear to stakeholders. ONS needs to learn lessons from this experience when engaging with user groups and when communicating its decision making, particularly where topics or opinions are contentious.

Meeting user needs

Uses of Census data are many and varied. Throughout Census planning and developments, engagement with a broad range of stakeholders has been required. We recognise the complex strategies which are put in place for ONS to engage and respond to different groups and the challenges that ONS faces in making decisions to best serve the public good when it comes to Census data and statistics. This was demonstrated during its recent outputs consultation where ONS reached out to a range of interest groups – such as the gypsy and traveller community, carers and disability groups – building on the engagement with these groups during the Census collection operations.

As a more detailed example of this, we consider ONS’s work to deliver and respond to the needs of Welsh users and stakeholders and its commitment to the Welsh language in Wales. ONS has made clear in its White Paper, Help Shape Our Future: The 2021 Census of Population and Housing in England and Wales, its commitment to the Welsh language in Wales – for example through ensuring relevant legislative requirements are satisfied by making appropriate arrangements for enumeration, publicity and public engagement activities in Wales – and how it planned to engage with key Welsh stakeholders. We also recognise the involvement of Welsh Government in ONS’s working groups and committees involved in all levels of Census decision making.

As part of Census data collection, ONS delivered a variety of services with Welsh respondents in mind such as questionnaires, campaign materials, website and social media channels being produced in Welsh and English. When it comes to ongoing stakeholder engagement, ONS has shown it has made efforts to maintain, develop and foster key stakeholder relationships, for example ONS has a dedicated Census Advisory Group for Wales.

We are aware however that certain stakeholder issues may not have been picked up as effectively as needed through these engagement routes. For example, we understand issues were raised with ONS about Welsh language addresses on paper and electronic questionnaires. These issues were not fully taken into consideration until they were raised in writing with ONS, at which point it was too close to Census Day to be acted upon. We were pleased to hear therefore that actions such as the terms of reference for the Census Advisory Group for Wales are being reviewed as this helps ensure both Welsh representatives and ONS reflect and make most effective use of this engagement forum.

We have been told there is a lot of interest currently in Census outputs in the Welsh language. We understand ONS is currently running a discovery piece on the need for Welsh language outputs and we are also very pleased to hear ONS has created a role for a Welsh language champion. One area the new champion will be leading on is to establish a policy for addressing in Welsh across the ONS. These are further positive steps taken to address the needs of this user group and ONS should be transparent and clear on how it will meet their needs. Overall it is positive to see ONS taking steps to strengthen its efforts to effectively engage with and respond to the needs of Welsh stakeholders.

Census user engagement and consultation is one of the largest scale exercises of its type in the collection and production of data and statistics in the UK. With such a wide and varied set of users of Census data, ONS needs to engage with user groups with different requirements and interests. This includes from the special interest groups or from a topic focussed perspective or when, for example, considering the needs of users with different levels of expertise or accessibility requirements.

Communicating plans for Census outputs

Throughout our user engagement, in both phases of the assessment, there has been strong interest from users to understand what and when Census data will be available to them. Users told us that they need clarity and transparency on the release schedule, as well as the outputs catalogue, so that they can plan their own outputs or broader work accordingly. While we understand that some of this information will be dependent on the outcomes of its outputs consultation, we recognise the demand for this information to support those who intend to use Census data. ONS published its proposed outputs release plans in July 2021 and expects to publish full details of its outputs release schedule in February 2022.

For Census 2021, ONS originally stated that it aims to make Census information flexible, timely and accessible, and specifically that it would aim to disseminate national and local authority-level estimates for England and Wales within 12 months of Census and all other estimates within 24 months of Census. On 12 October 2021, ONS informed users, primarily through a blog piece in National Statistical, that due to improvements to its quality assurance arrangements for Census data, first outputs will now be available in late spring 2022.

Producing timely and accurate data from the Census is vital to ensuring high public value. We recognise the improvements to ONS’s quality assurance processes and how this reflects ONS’s commitment to quality, although this has impacted on the release schedule for Census outputs.

Keeping the general public and users informed

Efforts to engage with Census data users do not receive the same investment and effort as the engagement activity with the public ahead of Census day, and arguably rightly so. ONS offers a range of engagement opportunities for Census users and stakeholders and, with more social media activity, may reach out to a broader audience base. ONS has told us that it is developing a package of communications around Census outputs, building on the campaign during the running of Census, which it hopes to reach a wide range of groups including the wider citizen audience. Aiming to reach this broader audience base not only helps promote and explain the value of Census data and statistics but also helps to acknowledge the importance of respondent’s participation across England and Wales during 2021.

ONS’s Census website gave the general public easily accessible information about Census, supporting their participation during the collection phase, and has been updated with links to the recent outputs consultation. Detailed information on Census research, developments and plans are hosted on Census webpages on the existing ONS website. During Phase 1 of this assessment, we found that there were weaknesses in the ease of navigation of ONS’s webpages and that ONS needed to do more to improve the accessibility of key information. ONS has made some improvements to its Census webpages however it can still be difficult to find materials. For example, if looking for key information on quality and methods, it may not be obvious that that information is held under the Census Outputs tab – subheading ‘Our output plans for the data from Census 2021’ – rather than, say, the Census design tab – subheading ‘How we run, process and quality assure Census 2021’. Some of the tools used to communicate progress and future plans through the webpages, such as the Census News webpage and Milestones webpages, have not been consistently kept up to date although we do recognise that ONS has more recently refreshed this content.

Having accessible and easily findable information supports the appropriate use of data and statistics. The current ONS Census webpages contain a wide range of materials, research, plans, and reports, with the aim of providing information to aid with transparency throughout its planning and development stages. However currently it is not achieving those aims as effectively as one might expect.

A continuous improvement approach to Census

Innovation and improvement in Census outputs

ONS has acknowledged that its past approach of trying to meet a range of user needs through a set of ‘standard tables’ has limitations. To address these limitations for Census 2021, ONS is developing a more flexible dissemination approach for multivariate data, that has automated disclosure control methods at its heart, by creating a Flexible Table Builder. This tool will allow users to generate bespoke tables in a timely and automated fashion, whilst still ensuring the data are subject to disclosure control. The Flexible Table Builder will involve three types of disclosure control methods: record swapping, cell key perturbation and disclosure checks. The development of this tool should significantly improve the flexibility and timely outputs offering from ONS for Census 2021 when this is realised.

As well as improving the accessibility, ONS has focused efforts on supporting public understanding of Census data through data visualisation. ONS presented its plans for new interactive outputs at the Royal Statistical Society (RSS) Conference 2021. These proposed outputs include mapping tools which allow users to aggregate data for non-GSS geographies, and ‘data-driven templates’ which can be embedded in a ‘scrollytelling’ release[1]. ONS has demonstrated a commitment to innovation and improvement through its plans for enhanced outputs tools and reporting media for Census 2021, which should serve to bring greater clarity, insight and accessibility for data users.

Scrollytelling is a new format for the ONS digital content team, being developed for Census data, where users can scroll to zoom into different levels of geographies and see how the data and narrative differs at each level.

Retaining lessons learned

At the end of Census processing and outputs, ONS will undertake a full assessment of the benefits of Census as it did for Census 2011. The assessment team spoke with ONS about its current thinking regarding realising the benefits across ONS and the wider Government Statistical Service of what it has learned from Census. We were encouraged to hear about ONS’s plans to establish a legacy working group to take this forward. This group will consider the wide range of developments across services, people and processes involved in Census, many of which already are recognised as having been successful. For example, the electronic questionnaire and digital delivery, branding and engagement, and field force management.