Annex 1: Preliminary findings from the first phase of this assessment

Finding 1

Some users have reported difficulties in locating certain documents on the Census offices’ websites. The design and navigation through these websites vary notably between Census offices and may provide a barrier to the easy access of information for users.

Actions taken by NISRA to address findings:

• NISRA has reviewed its Census webpages and implemented a similar structure, with menu headings, in line with the National Records of Scotland (NRS) webpages. NISRA has added content to these webpages, some in response to other assessment findings such as information on assurance mechanisms and a UK Census data page. NISRA does this in compliance with the Web Content Accessibility Guidelines.
• Much of the new content was developed in consultation with the other UK Census offices, with the aim of using consistent layout and wording where possible to provide a common experience for UK-level users who access Census 2021 information from the NISRA, Scotland’s Census and the Office for National Statistics (ONS) websites.
• The Census webpages continue to be easily accessible from the main NISRA homepage.
• The team told us about the development of an outputs dissemination platform which will be accessed through the NISRA website. This is in its early stages and NISRA told us that it has consulted with users on how they would prefer to access this platform. The platform will be developed more widely for users of other NISRA statistics and data.

OSR’s evaluation of evidence:

• NISRA has sought to meet the aims of the recommendation and should continue to consider coherence and user experience as more content is published on its website. The steps the team has taken to implement a clearer structure have been successful and the structure now lends itself to accommodate new and future published content.
• It is clear the team are working with colleagues in NRS and ONS to consider common user experience and share good practice. The team has helpfully used common language for menu headings with NRS and ONS, which should make navigation to relevant materials for users more straightforward.
• NISRA is still considering the development of the outputs dissemination platform and has sought user feedback on this. We encourage the team to continue this engagement as it develops this platform.

Finding 2

There has been much public debate relating to proposals for Census questions and the question development process across Census offices. Census offices must ensure their actions and decisions are visibly taken with the honesty, integrity and independence necessary to deliver data and statistics that are of high quality and serve the public good.

Actions taken by NISRA to address findings:

• NISRA has provided a summary of topic reports and consultations which were taken account of in its question development. This includes the recent publication of an update on UK Armed Forces Veterans topic report.
• The assessment team understands that NISRA has also held meetings with user groups interested in topics including the sex question, main language question and health conditions question as part of its development process. We understand that NISRA continues to engage with users and stakeholders on certain topic areas.
• Alongside this, committee hearings are available for viewing online after NISRA appeared before the Northern Ireland Assembly Committee for Finance on 18 March 2020 to discuss the presentation of evidence in relation to the Census Order (Northern Ireland) 2020.

OSR’s evaluation of evidence:

• NISRA continues to publish materials to evidence its decision-making processes on Census questions.
• We note the update on the UK Armed Forces Veteran topic report which outlines the research leading to the decision to not include a veteran question in the 2021 Census in Northern Ireland. However, the assessment team considers that NISRA should do more to alert users and interested parties of the any new Census materials when they are released.
• NISRA should continue to engage with stakeholders particularly in relation to areas of contention, meet any commitments it has made, and seek to provide answers or explanations on areas of concern, in a transparent and open way

Finding 3i

Census offices could benefit from working together more closely to better engage with users and stakeholders of UK Census data and statistics who have views and insights that are relevant across Census offices. UK Census users and stakeholders may be unnecessarily overburdened if they are required to submit multiple feedback or consultation responses on similar topic areas across Census offices.

Actions taken by NISRA to address findings

Since the publication of the assessment report, NISRA, working with the other Census offices, has:

• established a UK Census Data working group. The working group is specifically looking at the assessment finding on UK Census data users. The group has been considering options such as UK wide events.
• added a new UK Census data webpage, inviting UK Census data users to contact the Census team. The webpage should serve as a clear route to publish materials of particular interest to UK users.
• attended cross-office user events. While this already happened, steps have been taken to be more visible at these events through ensuring people from other offices are introduced to attendees.

OSR’s evaluation of evidence

We consider the UK Census data working group to be an extremely positive initiative put in place to address this finding. We are encouraged by NISRA’s engagement with this group and the steps taken so far.

• The new UK Census data webpage is a positive step and having tailored content for the specific group of users is a valuable addition. This should hopefully encourage engagement with UK Census data users so NISRA and the other Census offices can better understand their data needs.
• We recognise the barriers that Census offices may face in identifying and engaging with this group of users. The assessment team is hopeful that the continuing efforts of the working group will help with this. The working group should look to increase the visibility of its activities and communications and draw on the expertise across the wider GSS in exploring new and creative ways to reach users groups that are harder to engage with.

Finding 3ii

Published plans, research and topic reports from the individual Census offices are not always sufficiently clear on how country-specific proposals then relate to UK users’ needs and about the impact of these decisions on the availability of on harmonised UK data.

Actions taken by NISRA to address findings:

• NISRA, with the other Census offices, released a progress update on the Conduct of the 2021 Censuses in the UK in January 2020, reflecting the position as of November 2019. The statement of agreement on the conduct of the Censuses sets out the principles that the three Census offices will work together on to make sure the 2021 Censuses are successful. NISRA told us that there are plans to release a further updated position at the end of this year.
• The UK Census Data working group, which NISRA is part of, is compiling a descriptive list of all the harmonisation working groups and activities contributed to by the UK Census offices to capture the variety of work that goes into UK harmonisation. The UK Census data working group plans to publish this information and the Census offices are considering whether the most appropriate vehicle to do so is alongside the planned update on the conduct of the Censuses.
• Some high-level summary information on UK harmonisation is included on the new UK Census Data webpage.

OSR’s evaluation of evidence:

• There is an array of groups in place which consider harmonisation as part of Census plans and development. Putting information about these groups in the public domain will offer greater transparency to users and stakeholders.
• When collating this information, if feasible, examples of how decision making has taken account of the needs of UK Census users should be included.
• NISRA should be transparent in how it meets the needs of users interested in UK Census outputs in future publications on harmonised questions, outputs and methods.
• Due to the decision by Scottish Ministers to delay the Census in Scotland, it is even more important NISRA must work with the other Census offices to explain the impacts of this to UK Census data users.

Finding 3iii

Proposals from Census offices on Census questions will be subject to relevant legislative processes in finalising arrangements for Census in 2021.

Actions taken by NISRA to address findings:

• In response to this finding NISRA has described the collaborative approach to question development between the three Census offices.
• NISRA has shared the plans to review and produce information on harmonisation of Census questions when legislative processes are finalised and will likely publish this in 2021.

OSR’s evaluation of evidence:

• The assessment team recognises the commitment of NISRA and the other Census offices to publish information on harmonisation for users.

Finding 4

Census offices have a work programme scheduled to establish and evaluate the data sources that will be used to support Census activities. In some cases, we are not fully assured that the necessary conversations with all the various data suppliers are taking place to inform the understanding of the nature, operational context and overall quality of the sources to be used.

Actions taken by NISRA to address findings:

• NISRA has shared information and examples of how it assures the quality of administrative data used in Census in its evidence report. It describes how NISRA records data quality issues internally and feeds back to data supplier to improve data, how it assesses quality through comparing with other sources, how it engages with data suppliers, and how NISRA builds its knowledge and awareness of data quality issues through engaging with the wider statistical network – for example involvement in the National Statistician’s Quality Review on data linkage and at the International Census Forum.
• The assessment team held a data sources session with the Census team in March 2020 to hear more about the administrative data and how the team assures the quality these data.
• OSR also ran a Quality Assurance of Administrative Data (QAAD) session with the three Census offices in May 2020, to respond to queries raised on our expectations on for quality assurance of administrative data as part of Census. NISRA attended and engaged positively in that session.
• NISRA has told the assessment team that it has been hampered in publishing information on its use of administrative data by the impact of the COVID-19 pandemic. The administrative data team’s focus has been on maintaining support for Census 2021 data processing, mainly through further development and refinement of the administrative data-based person spine.

OSR’s evaluation of evidence

The assessment team welcomes the openness of the Census team in NISRA to discuss and seek advice regarding its use of administrative data.

• At the data sources session, the assessment team held with the Census team in March, NISRA demonstrated awareness of the strengths and limitations of each of its data sources, and of the quality assurance processes for each source. NISRA told us how it regularly engages with suppliers and seeks to understand the quality of their data.
• The assessment team recognises the work of NISRA on appropriate use of administrative data and recognises the resource challenges that NISRA has faced over the period of the pandemic.
• Being transparent and providing users and stakeholders with assurances on its judgements is a key aspect of the Code of Practice for Statistics. We consider that information on data sources used in the Census and NISRA’s judgement on quality and appropriateness for use should be made available in the public domain. The level of detail should be proportionate to level of assurance required, and NISRA should consider publishing provisional information and building on this as its research and understanding continues to develop.

Finding 5

Census offices are developing their quality management approaches and developing various methods to support Census operations, data processing and outputs. This includes addressing potential biases between online and paper collection modes. However, the timing for publication of methodology documentation and quality assurance arrangements is unclear.

Census offices should make information on the methodology and quality assurance arrangements available to users at the earliest opportunity.

[/panel]

Actions taken by NISRA to address findings

• There is currently limited information on NISRA’s data collection operations, methodology, and quality assurance arrangements publicly available. However, high-level descriptions of the data collection processes and quality assurance can be found in the 2021 Census Proposals document, published in April 2019. Furthermore, in August 2020 NISRA published its Population Definitions for Northern Ireland Census 2021.
• NISRA states in its evidence report that it is currently developing its quality assurance and data collection approaches for Census 2021 and that intelligence gained during the 2019 rehearsal will be a key element in refining the methods. NISRA recognises the importance of developing its quality assurance approach for Census 2021 and additional resource has been assigned to progress work on its overall quality assurance strategy.
• NISRA told the assessment team that it expected to hold public information events during winter 2020/21, at which Census users, stakeholders and the general public will be able to learn about the planned data collection operation.

OSR’s evaluation of evidence

• The assessment team recognises that NISRA is still refining and developing methods following its Census rehearsal. However, there remains very limited information on methods or quality assurance arrangements in the public domain.
• We encourage NISRA to publish more information on their data collection, methodology and quality assurance arrangements at the earliest opportunity. In the interim, NISRA should consider what materials could be made public to inform and assure users of the plans and processes to ensure Northern Ireland’s Census data will be of sufficient quality.
• The assessment team would like to explore methods being employed by NISRA for Census in more detail to cover how methods have been developed, what methods have been selected and why NISRA consider these suitable for use. We are hoping to hold workshops on areas such as census coverage adjustment and statistical disclosure control in the coming months

Finding 6

Users told us that they were unclear at what points certain decisions will be made or when certain information will be made available. While planning documents are published, there is no easily accessible high-level plan of key Census milestones.

Actions taken by NISRA to address findings

• NISRA has now added information on census milestones to its website. This covers areas such as legislative process, operational aspects, outputs, and this assessment process. The status is provided for each milestone and there are links to supporting information.

OSR’s evaluation of evidence

• The assessment team is pleased to see that this overview information is now available to users and interested parties.
• NISRA should maintain this information providing further detail or updates on timings as appropriate.

Finding 7

Use of internal review, audit and other independent measures to evaluate the effectiveness of processes is important across all three Census offices and provides additional safeguards.

Actions taken by NISRA to address findings

• NISRA has published information on its webpage on the assurance mechanisms in place to underpin Census delivery. Information provided describes internal oversight through Census Office Senior Management Team and the Census Oversight Board, and external assurance via Census Assurance Panel and Demographic Statistics Advisory Group.
• The minutes from the last Demographic Statistics Advisory Group held in January 2020 have yet to be published. After speaking with us, NISRA has confirmed that the minutes from the last two DSAG meetings have been approved for publication on the relevant section of the NISRA website and that this is expected to happen soon.
• On 14 August 2020, NISRA published its 2019 Census Rehearsal Evaluation Report which summarised how its tests of processes and systems performed in areas such as stakeholder engagement, address register and field operation.

OSR’s evaluation of evidence

• NISRA has published the information required which allows users to better understand the broader assurance and audit activity in place to evaluate processes.
• It is important that minutes and papers from meetings like that of the Demographic Statistics Advisory Group are uploaded in a timely manner to ensure they are relevant, and users can access them when needed. NISRA should publish the January 2020 Demographic Statistics Advisory Group minutes as soon as possible. We note the recently published rehearsal evaluation report. It highlights areas of operational success as well as areas which will require improvement for Census 2021.
• We would be particularly keen to engage further with NISRA on its plans to address low student response rates which were evident in the rehearsal.