Annex 1: Preliminary findings from the first phase of this assessment

Finding 1

Some users have reported difficulties in locating certain documents on the Census offices’ websites. The design and navigation through these websites vary notably between Census offices and may provide a barrier to the easy access of information for users.

• Actions taken by ONS to address findings ONS has added new content to its Census 2021 webpages, some in response to other assessment findings such as information on assurance mechanisms. ONS told us it has developed much of this new content working in collaboration with Northern Ireland Statistics and Research Agency (NISRA) and National Records of Scotland (NRS), so common layouts and wording were used where possible.

ONS has shared with us its plans to increase the accessibility of its current webpages which include employing a user-centric rather than topic-based approach to navigation through the webpages, and alignment with other Census offices where possible.

ONS recognises the need for harmonised content where possible to achieve a common user experience, with increased signposting and cross-site linking.

ONS also told us about the development of its outputs dissemination platform which will be integrated into its existing online presence. Any developments and solutions will be delivered in line with Government Digital Service standards. Separate developments are being taken forward by NRS and NISRA and, while the platforms may have differences, ONS has said it will aim to align web design and build similar user journeys across the different systems.

• OSR’s evaluation of evidence The plans ONS outlined to improve its Census webpage are positive and take account of the issues highlighted through this assessment, although it is disappointing more could not be achieved or implemented to date. For example, within its ‘Census Transformation’ page there is content relating to Census 2021 and on the Admin Data Census Project; reorganisation and grouping of content with more intuitive naming would make it easier for users to locate materials they are interested in.

ONS should implement its plans for Census webpage improvements in the coming months and involve user testing as it considers navigation and accessibility. This will be more important as additional content is published on the website.

ONS’s plans for the development of the outputs dissemination platform are positive.

Finding 2

There has been much public debate relating to proposals for Census questions and the question development process across Census offices. Census offices must ensure their actions and decisions are visibly taken with the honesty, integrity and independence necessary to deliver data and statistics that are of high quality and serve the public good.

• Actions taken by ONS to address findings ONS has published additional question development materials since the completion of the first phase of the assessment, these include
  1. Question and questionnaire development overview for Census 2021 (published March 2020) which provides users with information of the development processes including approach to question development, research methods, and criteria to evaluate question design.
  2. Summary of testing for Census 2021 (published March 2020) which lists all testing conducted between 2016 and 2020 that informed the question design for Census 2021 in an excel spreadsheet.

• Question development reports (published June 2020) which explain the research, testing and evaluation carried out on question design on topics such as sex and gender identity, ethnic group tick-box, national identity, ethnic group, language and religion, armed forces, qualifications and housing.

Materials on question development are all published on a common webpage.

ONS has kept the assessment team informed of activity, particularly in relation to the guidance for the sex question in the Census. The assessment team understands that there has been ongoing engagement with users and stakeholders on this subject and that ONS has committed to undertake further testing and engagement with data users.

• OSR’s evaluation of evidence ONS has published a range of materials to evidence its decision-making processes on Census questions. The question development overview, which provides details of its research methods, questionnaire design and evaluation criteria, is particularly useful in this regard.

The summary of all testing undertaken by ONS on Census questions and questionnaires is a helpful resource. This should be kept updated and would be further enhanced if it provided a link through to testing outcomes, such as topic reports where feasible.

While a range of materials has been published, the assessment team considers that more could be done to alert users and interested parties of new content when it is released.

The assessment team thinks it essential for ONS to consider the concerns raised by users during its further testing and research on the guidance on the sex question and consider the impact of data quality on the analysis of small sub-groups of the population.

ONS should continue to engage with stakeholders particularly in relation to areas of contention, meet any commitments it has made, and seek to provide answers or explanations on areas of concern, in a transparent and open way.

Finding 3i

Census offices could benefit from working together more closely to better engage with users and stakeholders of UK Census data and statistics who have views and insights that are relevant across Census offices. UK Census users and stakeholders may be unnecessarily overburdened if they are required to submit multiple feedback or consultation responses on similar topic areas across Census offices.

• Actions taken by ONS to address findings Since the publication of the assessment report ONS, working with the other Census offices, has:
  1. established a UK Census Data working group. The working group is specifically looking at the assessment finding on UK Census data users. ONS has taken a lead role in this work and has a member of staff in place who has clear responsibilities for engagement with UK users. The group has been considering options such as UK wide events.
  2. added a new UK Census data webpage, inviting UK Census data users to contact the Census team. The webpage should serve as a clear route to publish materials of particular interest to UK users.

• attended cross-office user events. While this already happened, steps have been taken to be more visible at these events through ensuring people from other offices are introduced to attendees.

• OSR’s evaluation of evidence We consider the UK Census data working group an extremely positive initiative put in place to address this finding. We welcome the fact that ONS has taken the lead in driving forward activities focused on the UK Census data users.

The new UK harmonisation webpage is also a positive step and having tailored content for this specific group of users is a valuable addition. ONS should make this webpage more prominent and easier for UK Census data users to find.

We recognise the barriers that Census offices may face in identifying and engaging with this group of users. The assessment team is hopeful that the continuing efforts of the working group will help with this. The working group should look to increase the visibility of its activities and communications and draw on the expertise across the wider GSS in exploring new and creative ways to reach users groups that are harder to engage with.

Finding 3ii

Published plans, research and topic reports from the individual Census offices are not always sufficiently clear on how country-specific proposals then relate to UK users’ needs and about the impact of these decisions on the availability of on harmonised UK data.

• Actions taken by ONS to address findings ONS, with the other Census offices, released a progress update on the Conduct of the 2021 Censuses in the UK in January 2020, reflecting the position as of November 2019. The statement of agreement on the conduct of the censuses sets out the principles that the three Census offices will work together on to make sure the 2021 Censuses are successful. ONS told us there are plans to release a further updated position at the end of this year.

ONS, in its report How the Office for National Statistics is ensuring the 2021 Census will serve the public published as part of this assessment, set out the involvement of cross-Census offices in various groups and committees. This will be further supplemented by the collation of information on the range of harmonisation working groups across the range of Census activities. ONS told us it plans to publish this information and is considering whether the most appropriate vehicle to do so is alongside the planned update on the conduct of the Censuses.

Some high-level summary information on UK harmonisation is included on the new UK harmonisation webpage.

• OSR’s evaluation of evidence ONS has frameworks in place to facilitate decision-making which takes account of the views and opinions from across the Census offices. There is an array of groups in place which consider harmonisation as part of Census plans and development. Putting information about these groups in the public domain will offer greater transparency to users and stakeholders.

When collating this information, if feasible, examples of how decision making has taken account of the needs of UK Census users should be included.

ONS should also seek to engage with UK Census users on their response to how it has taken account of their needs.

ONS should be transparent in how it meets the needs of users interested in UK Census outputs in future publications on harmonised questions, outputs and methods.

Due to the decision by Scottish Ministers to delay the Census in Scotland, it is even more important ONS must work with the other Census offices to explain the impacts of this to UK Census data users.

Finding 3iii

Proposals from Census offices on Census questions will be subject to relevant legislative processes in finalising arrangements for Census in 2021.

• Actions taken by ONS to address findings In response to this finding, ONS has described the collaborative approach to question development between the three Census offices.

ONS has shared the plans to review and produce information on harmonisation of Census questions when legislative processes are finalised and will likely publish this in 2021.

ONS has included question harmonisation content in its recent question development reports, which provides a summary of the approaches taken to questions across the three Censuses.

• OSR’s evaluation of evidence The assessment team recognises the commitment of ONS and the other Census offices to publish information on harmonisation for users.

Finding 4

Census offices have a work programme scheduled to establish and evaluate the data sources that will be used to support Census activities. In some cases, we are not fully assured that the necessary conversations with all the various data suppliers are taking place to inform the understanding of the nature, operational context and overall quality of the sources to be used.

• Actions taken by ONS to address findings ONS has provided an overview of how it coordinates its requirements and arranges the supply of data from other government departments or organisations, how it manages relationships with data suppliers and how it records information on data quality.

In its evidence report, ONS talks about its involvement in the UK Population and Censuses Strategic Group – which considers the use of administrative data in the 2021 Census; the Administrative Data Research Group – to share information and best practice on the integration of administrative data in the Census process; and collaboration with other national statistical institutes. In the report ONS also mentions its work with Ministry of Defence (MoD) and Valuation Office Agency (VOA) on data for Census use. Further information on the use of MoD data can be found on the Armed forces community question development webpage. ONS has also recently published materials on number of rooms using VOA data including a summary of the quality assurance undertaken and its judgements on the quality of this data.

OSR ran a Quality Assurance of Administrative Data (QAAD) session with the three Census offices in May 2020, to respond to queries raised on our expectations for quality assurance of administrative data as part of Census. ONS attended and engaged positively in that session. OSR also spoke with ONS teams in July 2020 to better understand the processes and practices with regard to the acquisition and preparation of nonsurvey data through the Data as a Service (DaaS) division, ONS’s administrative data strategy for Census, and the quality framework being developed for use of administrative data in Census processing.

ONS also plans to publish a paper proposing a quality framework and strategy for measuring the quality of administrative data used in Census 2021.

• OSR’s evaluation of evidence The assessment team appreciates the openness of ONS teams to discuss and seek advice regarding its use of administrative data.

In discussions on the function and role of DaaS division, ONS spoke of changes to its operating model and its ambitions for improvements. We are encouraged ONS is reflecting and looking to enhance its processes. For example, ONS told us it is considering how it can improve the quality of the source data through greater collaboration and communication of its understanding of quality considerations with data suppliers.

We note the future publication of the paper on measuring the quality of administrative data used in Census. It will be a positive step to provide users and stakeholders with assurances of how ONS consider data quality when using administrative data.

We also note the recent publication of information on VOA’s property attribute administrative data.

Being transparent and providing users and stakeholders with assurances on its judgements is a key aspect of the Code of Practice for Statistics. We consider that information on data sources used in the Census and ONS’s judgement on quality and appropriateness for use should be made available in the public domain. ONS should continue to publish information on its use of data sources and, where appropriate, consider publishing provision information and building on this as its research and understanding continues to develop. ONS should ensure this information is easily accessible on its website.

Finding 5

• Census offices are developing their quality management approaches and developing various methods to support Census operations, data processing and outputs. This includes addressing potential biases between online and paper collection modes. However, the timing for publication of methodology documentation and quality assurance arrangements is unclear. Census offices should make information on the methodology and quality assurance arrangements available to users at the earliest opportunity.
• Actions taken by ONS to address findings ONS published a document in January 2020 setting out its approach and processes for assuring the quality of 2021 Census data. This included an overview of its quality assurance strategy and further details on the processes involved – from the address frame through data collection and imputation, to published outputs – and how it manages those.

ONS has committed to update its methodology webpage and include links to census methods work from the GSS methods webpages.

ONS has also started to publish the papers and minutes from the Census Methodological Assurance Review panel on the UK Statistics Authority website.

The assessment team has held sessions in July 2020 with ONS to discuss census methods – what methods have been selected and why ONS consider these suitable for use. These sessions included discussions on item imputation, statistical disclosure control, and census coverage adjustment.

• OSR’s evaluation of evidence The assessment team recognises ONS’s commitment to update the Census methodology webpages as currently the information contained there is sparse.

We are pleased to see ONS publishing methods papers reviewed by the external assurance panel; this brings transparency to methods development and assurance over the rigor being taken in that regard. ONS could better alert users and interested parties to this information, not least by linking to these papers from the Census methodology webpage.

Based on the assessment team’s methods sessions with ONS, we consider that ONS is taking a rigorous approach to its methods development. The assessment team is keen to continue conversations with ONS to understand how any decisions or approaches to methods may need to take account of behaviour changes of the UK population as a result of the COVID-19 pandemic.

However, as many methods approaches for Census are due to be signed off by the end of the year, we would again look for ONS to make its development work more visible and accessible to expert audiences. The assessment team also considers that ONS should, when publishing finalised methods information, make clear where approaches may differ from other UK Census offices and the reasons why that decision has been made.

We welcome the publication of the quality assurance document. This is an important document to assure users and stakeholders of the processes and strategy in place. It is well set out and readable for users who may not have an in-depth knowledge of Census processes.

Finding 6

Users told us that they were unclear at what points certain decisions will be made or when certain information will be made available. While planning documents are published, there is no easily accessible high-level plan of key Census milestones.

• Actions taken by ONS to address findings ONS has now added information on census milestones to its website. This covers areas such as legislative process, operational aspects, outputs, and this assessment process.
• OSR’s evaluation of evidence The assessment team is pleased to see that this overview information is now available to users and interested parties.

ONS should maintain this information providing further detail or updates on timings as appropriate. ONS should also consider adding links to information on completed milestones where this is possible, such as linking to rehearsal evaluation materials once these are published.

Finding 7

Use of internal review, audit and other independent measures to evaluate the effectiveness of processes is important across all three Census offices and provides additional safeguards.

• Actions taken by ONS to address findings ONS has published information on its webpage on the assurance mechanisms in place to underpin Census delivery. Information provided described both internal and external assurance processes and describes at a high level how these difference activities are managed and also mentions implementing improvements, such as lessons learnt through 2019 rehearsal.
• OSR’s evaluation of evidence ONS has published the information required which allows users to better understand the broader assurance and audit activity in place to evaluate processes.

ONS should continue to be transparent on outcomes of evaluation and audit activities where this is feasible by publishing materials such as Census rehearsal evaluation reports.